Falco、Prometheus、Grafana、Dockerを使用してサーバーインフラストラクチャクラウドを保護する

私は最近、コンテナとアプリケーションを機能させ続ける方法を探していました。具体的には、構成の異常を検出することに興味がありました。少し調べた後、私はFalcoに出くわしました。

私が見つけたのは、多くの機能と優れたドキュメントを備えた非常に完全なオープンソースプラットフォームでした。だから私はあなたと私の経験を共有したいと思いました。

この記事では何を取り上げますか?

  • イベント(異常/違反)を監視するホストにFalcoエージェントをインストールする方法
  • Falcoを調整して誤検知を減らし、本当に必要な情報を取得する方法
  • Prometheusを使用して、輸出業者とスクレーパーの助けを借りて、Falcoイベントを中央の場所に収集する方法。
  • 最後に、視覚化とアラートのためにスクレーパーをGrafanaに接続する方法

このチュートリアルには何が必要ですか?

  • Linuxがインストールされている1つまたは複数のマシン。仮想マシンが機能するはずです。
  • Docker、RPM、およびsystemdプロセスをインストール/セットアップできるようにするには、スーパーユーザー権限が必要です。
  • Dockerコンテナーを使用するため、Dockerの基本的な知識が必要です。
  • 構成をテストおよび改善するためのスクリプトをいくつか作成するため、Python/Bashの実用的な知識。

最後に、次の各コンポーネントをセットアップできるようになります。

falco_monitoring

恐れることはありません。ドキュメントへのリンクと、これらの各タスクの詳細な説明を提供します。

目次

  1. ファルコとは?
  2. Falcoのインストール方法
  3. 基本構成
  4. デフォルト設定をテストする方法
  5. デフォルトは必ずしも良いとは限りません
  6. Falco統合
  7. 詳細はこちら

ファルコとは?

このツールを説明する最良の方法は、ツールで何ができるかを学ぶことです

Falcoは、Linuxシステムコールの実行に関連するすべての動作を検出して警告することができます。

Falcoアラートは、特定のシステムコールとその引数の使用、および呼び出しプロセスのプロパティによってトリガーできます。たとえば、Falcoは、以下を含むがこれらに限定されないインシデントを簡単に検出できます。

  • Kubernetesのコンテナまたはポッド内でシェルが実行されています。
  • コンテナが特権モードで実行されているか、ホストから/procなどの機密パスをマウントしています。
  • サーバープロセスは、予期しないタイプの子プロセスを生成しています。
  • / etc/shadowなどの機密ファイルの予期しない読み取り。
  • デバイス以外のファイルが/devに書き込まれます。
  • lsなどの標準のシステムバイナリは、アウトバウンドネットワーク接続を確立しています。
  • 特権ポッドがKubernetesクラスターで開始されます。

Falcoのインストール方法

RPMを使用してFalcoをインストールします(apt-get、さらにはDockerコンテナーにも同様の手順があります)。私の場合、ネイティブインストールが最適であり、RPMによって非常に簡単に実行できると感じました。

[josevnz@macmini2 ~]$ sudo -i dnf install https://download.falco.org/packages/rpm/falco-0.31.1-x86_64.rpm
Last metadata expiration check: 2:53:53 ago on Sun 01 May 2022 04:13:09 PM EDT.
falco-0.31.1-x86_64.rpm                                                                                                                                                                                                       1.7 MB/s |  12 MB     00:07    
Dependencies resolved.
==============================================================================================================================================================================================================================================================
 Package                                                          Architecture                                      Version                                                                     Repository                                               Size
==============================================================================================================================================================================================================================================================
Installing:
 falco                                                            x86_64                                            0.31.1-1                                                                    @commandline                                             12 M
Installing dependencies:
 dkms                                                             noarch                                            2.8.1-4.20200214git5ca628c.fc30                                             updates                                                  78 k
 elfutils-libelf-devel                                            x86_64                                            0.179-2.fc30                                                                updates                                                  27 k
 kernel-devel                                                     x86_64                                            5.6.13-100.fc30                                                             updates                                                  14 M

Transaction Summary
==============================================================================================================================================================================================================================================================
Install  4 Packages

Total size: 26 M
Total download size: 14 M
Installed size: 92 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): elfutils-libelf-devel-0.179-2.fc30.x86_64.rpm                                                                                                                                                                          253 kB/s |  27 kB     00:00    
(2/3): dkms-2.8.1-4.20200214git5ca628c.fc30.noarch.rpm                                                                                                                                                                        342 kB/s |  78 kB     00:00    
(3/3): kernel-devel-5.6.13-100.fc30.x86_64.rpm                                                                                                                                                                                1.9 MB/s |  14 MB     00:07    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                         1.8 MB/s |  14 MB     00:07     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                      1/1 
  Installing       : kernel-devel-5.6.13-100.fc30.x86_64                                                                                                                                                                                                  1/4 
  Running scriptlet: kernel-devel-5.6.13-100.fc30.x86_64                                                                                                                                                                                                  1/4 
  Installing       : elfutils-libelf-devel-0.179-2.fc30.x86_64                                                                                                                                                                                            2/4 
  Installing       : dkms-2.8.1-4.20200214git5ca628c.fc30.noarch                                                                                                                                                                                          3/4 
  Running scriptlet: dkms-2.8.1-4.20200214git5ca628c.fc30.noarch                                                                                                                                                                                          3/4 
  Running scriptlet: falco-0.31.1-1.x86_64                                                                                                                                                                                                                4/4 
  Installing       : falco-0.31.1-1.x86_64                                                                                                                                                                                                                4/4 
  Running scriptlet: falco-0.31.1-1.x86_64                                  

基本構成

非常に基本的な出力処理を行う場合を除いて、JSON出力を有効にします。

# Whether to output events in json or text
json_output: true

理由はすぐに明らかになります。

次に、Falcoエージェントを開始します。

[josevnz@macmini2 falco]$ sudo systemctl start falco.service 
[josevnz@macmini2 falco]$ sudo systemctl status falco.service 
● falco.service - Falco: Container Native Runtime Security
   Loaded: loaded (/usr/lib/systemd/system/falco.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-05-01 19:20:52 EDT; 1s ago
     Docs: https://falco.org/docs/
  Process: 26887 ExecStartPre=/sbin/modprobe falco (code=exited, status=0/SUCCESS)
 Main PID: 26888 (falco)
    Tasks: 1 (limit: 2310)
   Memory: 65.8M
   CGroup: /system.slice/falco.service
           └─26888 /usr/bin/falco --pidfile=/var/run/falco.pid

May 01 19:20:52 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 19:20:52 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 19:20:52 macmini2 falco[26888]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 19:20:52 macmini2 falco[26888]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 19:20:52 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 19:20:53 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 19:20:54 macmini2 falco[26888]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:

デフォルト設定をテストする方法

構成によっては、Falcoを起動した直後にイベントが発生する場合と発生しない場合があります。

[josevnz@macmini2 falco]$ sudo journalctl --unit falco --follow
-- Logs begin at Tue 2021-05-25 00:15:22 EDT. --
May 01 19:20:52 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 19:20:52 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 19:20:52 macmini2 falco[26888]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 19:20:52 macmini2 falco[26888]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 19:20:52 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 19:20:53 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 19:20:54 macmini2 falco[26888]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
May 01 19:20:55 macmini2 falco[26888]: Starting internal webserver, listening on port 8765

心配しないでください。Falcoに警告とアラートを記録させるいくつかのコマンドを実行します。これがどのように機能するかを見てみましょう!

特権コンテナを実行する方法

特権コンテナの使用は悪い習慣と見なされているので、このイベントがFalcoによって検出されるかどうかを見てみましょう。

[josevnz@macmini2 ~]$ docker run --rm --interactive --tty --privileged --volume /etc/shadow:/mnt/shadow fedora:latest ls -l /mnt/shadow
----------. 1 root root 1198 Nov 21 20:51 /mnt/shadow

そして私たちのファルコログ?

May 01 19:29:32 macmini2 falco[26888]: {"output":"19:29:32.918828894: Informational Privileged container started (user=root user_loginuid=0 command=container:bfb9637a47a6 kind_lumiere (id=bfb9637a47a6) image=fedora:latest)","priority":"Informational","rule":"Launch Privileged Container","source":"syscall","tags":["cis","container","mitre_lateral_movement","mitre_privilege_escalation"],"time":"2022-05-01T23:29:32.918828894Z", "output_fields": {"container.id":"bfb9637a47a6","container.image.repository":"fedora","container.image.tag":"latest","container.name":"kind_lumiere","evt.time":1651447772918828894,"proc.cmdline":"container:bfb9637a47a6","user.loginuid":0,"user.name":"root"}}

情報イベントとして表示されます。間違いなく注目すべきものの1つです。コンテナ上のアプリケーションに昇格された特権が必要かどうかを自問してください。

また、各メッセージにタグが付いていることにお気づきかもしれません。「mitre_*」のものに注意してください。これらは、攻撃と軽減に関するMitreAttackナレッジベースに関連しています。うん、あなたはそれらを読むのに少し時間を費やすでしょう。

/rootディレクトリにファイルを作成する方法

この例は、コンテナ内のボリュームと組み合わせたrootユーザーを悪用する方法を示しています...

[josevnz@macmini2 ~]$ docker run --rm --interactive --tty --user root --volume /root:/mnt/ fedora:latest touch /mnt/test_file
[josevnz@macmini2 ~]$ 

ファルコ反応:

May 01 19:32:02 macmini2 falco[26888]: {"output":"19:32:02.434286167: Informational Container with sensitive mount started (user=root user_loginuid=0 command=container:ef061174c7ef distracted_lalande (id=ef061174c7ef) image=fedora:latest mounts=/root:/mnt::true:rprivate)","priority":"Informational","rule":"Launch Sensitive Mount Container","source":"syscall","tags":["cis","container","mitre_lateral_movement"],"time":"2022-05-01T23:32:02.434286167Z", "output_fields": {"container.id":"ef061174c7ef","container.image.repository":"fedora","container.image.tag":"latest","container.mounts":"/root:/mnt::true:rprivate","container.name":"distracted_lalande","evt.time":1651447922434286167,"proc.cmdline":"container:ef061174c7ef","user.loginuid":0,"user.name":"root"}}

敏感なマウントが検出されました!

/binにファイルを作成して賭け金を上げましょう

さて、これを行うとしましょう:

[josevnz@macmini2 ~]$ sudo -i
[root@macmini2 ~]# touch /bin/should_not_be_here

ファルコはそれについてどう思いますか?

May 01 19:36:41 macmini2 falco[26888]: {"output":"19:36:41.237634398: Error File below a known binary directory opened for writing (user=root user_loginuid=1000 command=touch /bin/should_not_be_here file=/bin/should_not_be_here parent=bash pcmdline=bash gparent=sudo container_id=host image=<NA>)","priority":"Error","rule":"Write below binary dir","source":"syscall","tags":["filesystem","mitre_persistence"],"time":"2022-05-01T23:36:41.237634398Z", "output_fields": {"container.id":"host","container.image.repository":null,"evt.time":1651448201237634398,"fd.name":"/bin/should_not_be_here","proc.aname[2]":"sudo","proc.cmdline":"touch /bin/should_not_be_here","proc.pcmdline":"bash","proc.pname":"bash","user.loginuid":1000,"user.name":"root"}}

エラー、書き込み用に開かれたバイナリディレクトリ。良いキャッチ。

デフォルトは必ずしも良いとは限りません

Falcoがしばらく実行された後、どのような種類のイベントを無視し、どのイベントを調査したいのかを把握することをお勧めします。

最初のステップは、ペイロードでJSON形式を使用して、すべてのイベントのリストを取得することです。

sudo journalctl --unit falco --no-page --output=cat > /tmp/falco_json_lines.txt

'output = cat'は、タイムスタンプなしでメッセージペイロードを提供するようにjournalctlに指示します(心配しないでください。JSONメッセージ自体にタイムスタンプがあります)。

Starting Falco: Container Native Runtime Security...
Started Falco: Container Native Runtime Security.
Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
Falco initialized with configuration file /etc/falco/falco.yaml
Loading rules from file /etc/falco/falco_rules.yaml:
Loading rules from file /etc/falco/falco_rules.local.yaml:
Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Starting internal webserver, listening on port 8765
{"output":"19:29:32.918828894: Informational Privileged container started (user=root user_loginuid=0 command=container:bfb9637a47a6 kind_lumiere (id=bfb9637a47a6) image=fedora:latest)","priority":"Informational","rule":"Launch Privileged Container","source":"syscall","tags":["cis","container","mitre_lateral_movement","mitre_privilege_escalation"],"time":"2022-05-01T23:29:32.918828894Z", "output_fields": {"container.id":"bfb9637a47a6","container.image.repository":"fedora","container.image.tag":"latest","container.name":"kind_lumiere","evt.time":1651447772918828894,"proc.cmdline":"container:bfb9637a47a6","user.loginuid":0,"user.name":"root"}}
{"output":"19:32:02.434286167: Informational Container with sensitive mount started (user=root user_loginuid=0 command=container:ef061174c7ef distracted_lalande (id=ef061174c7ef) image=fedora:latest mounts=/root:/mnt::true:rprivate)","priority":"Informational","rule":"Launch Sensitive Mount Container","source":"syscall","tags":["cis","container","mitre_lateral_movement"],"time":"2022-05-01T23:32:02.434286167Z", "output_fields": {"container.id":"ef061174c7ef","container.image.repository":"fedora","container.image.tag":"latest","container.mounts":"/root:/mnt::true:rprivate","container.name":"distracted_lalande","evt.time":1651447922434286167,"proc.cmdline":"container:ef061174c7ef","user.loginuid":0,"user.name":"root"}}

これまでのところ面白そうですが、これはどうですか?

{"output":"23:04:10.609949471: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=openat command=bash fd.name=/home/josevnz/.bash_history-01112.tmp name=/home/josevnz/.bash_history-01112.tmp path=<NA> oldpath=<NA> host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.609949471Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":"/home/josevnz/.bash_history-01112.tmp","evt.arg.oldpath":null,"evt.arg.path":null,"evt.time":1651633450609949471,"evt.type":"openat","fd.name":"/home/josevnz/.bash_history-01112.tmp","proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.635602857: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=openat command=bash fd.name=/home/josevnz/.bash_history-01627.tmp name=/home/josevnz/.bash_history-01627.tmp path=<NA> oldpath=<NA> host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.635602857Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":"/home/josevnz/.bash_history-01627.tmp","evt.arg.oldpath":null,"evt.arg.path":null,"evt.time":1651633450635602857,"evt.type":"openat","fd.name":"/home/josevnz/.bash_history-01627.tmp","proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.635851215: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=rename command=bash fd.name=<NA> name=<NA> path=<NA> oldpath=/home/josevnz/.bash_history-01627.tmp host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.635851215Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":null,"evt.arg.oldpath":"/home/josevnz/.bash_history-01627.tmp","evt.arg.path":null,"evt.time":1651633450635851215,"evt.type":"rename","fd.name":null,"proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.661829867: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=rename command=bash fd.name=<NA> name=<NA> path=<NA> oldpath=/home/josevnz/.bash_history-01112.tmp host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.661829867Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":null,"evt.arg.oldpath":"/home/josevnz/.bash_history-01112.tmp","evt.arg.path":null,"evt.time":1651633450661829867,"evt.type":"rename","fd.name":null,"proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}

これは通常の/正当な操作です。このルールを強化するか、完全に削除する方法を見つけましょう。

まず、/etc/falco/falco_rules.yamlファイルを開き、ルール「シェル履歴の削除または名前変更」(前に見たJSON出力)を探します。

- list: docker_binaries
  items: [docker, dockerd, exe, docker-compose, docker-entrypoi, docker-runc-cur, docker-current, dockerd-current]

 macro: var_lib_docker_filepath
  condition: (evt.arg.name startswith /var/lib/docker or fd.name startswith /var/lib/docker)

- rule: Delete or rename shell history
  desc: Detect shell history deletion
  condition: >
    (modify_shell_history or truncate_shell_history) and
       not var_lib_docker_filepath and
       not proc.name in (docker_binaries)
  output: >
    Shell history had been deleted or renamed (user=%user.name user_loginuid=%user.loginuid type=%evt.type command=%proc.cmdline fd.name=%fd.name name=%evt.arg.name path=%evt.arg.path oldpath=%evt.arg.oldpath %container.info)
  priority:
    WARNING
  tags: [process, mitre_defense_evasion]

Falcoのルールは、公式ドキュメントで詳細に説明されています。この作品を見るだけで、いくつかのことに気付くでしょう。

条件について:

  1. 複雑なロジックをサポートし、
  2. のようなマクロvar_lib_docker_filepath
  3. のようなリスト(docker_binaries)
  4. のようなフィールドを持つ特別な変数proc.name

このファイルは変更しないことをお勧めします。/etc/falco/falco_rules.local.yaml代わりに、 :で必要なものをオーバーライドする必要があります。

# Add new rules, like this one
# - rule: The program "sudo" is run in a container
#   desc: An event will trigger every time you run sudo in a container
#   condition: evt.type = execve and evt.dir=< and container.id != host and proc.name = sudo
#   output: "Sudo run in container (user=%user.name %container.info parent=%proc.pname cmdline=%proc.cmdline)"
#   priority: ERROR
#   tags: [users, container]

# Or override/append to any rule, macro, or list from the Default Rules

例として、スーパーユーザー(root)の履歴が上書きされる場合は注意しますが、それ以外の場合は問題ないとします。最良の部分は、ルール全体をオーバーライドする必要がないことです。

したがって、元のルールには条件が追加されます。

- rule: Delete or rename shell history
  append: true
  condition: and user.name=root

ルールが適切に記述されていることを検証することは常に良い考えです。そのためには、元のルールとオーバーライドを一緒にチェックするようにFalcoに指示する必要があります。

[root@macmini2 ~]# falco --validate /etc/falco/falco_rules.yaml --validate /etc/falco/falco_rules.local.yaml 
Fri May  6 20:48:00 2022: Validating rules file(s):
Fri May  6 20:48:00 2022:    /etc/falco/falco_rules.yaml
Fri May  6 20:48:00 2022:    /etc/falco/falco_rules.local.yaml
/etc/falco/falco_rules.yaml: Ok
/etc/falco/falco_rules.local.yaml: Ok
Fri May  6 20:48:01 2022: Ok

# If the rules are OK, restart Falco
[root@macmini2 ~]# systemctl restart falco.service

Pythonで簡単なイベントエクスプローラーを作成する方法

どのルールがノイズであり、どのルールが有用であるかを理解するのは面倒であることにおそらく同意するでしょう。

このデータを正規化する必要があり、次のようなPythonスクリプトを使用します。

  • 非JSONデータを削除する
  • タイムスタンプなしでイベントタイプを集約します
  • いくつかの集計統計を生成して、システムで最も頻繁に発生するイベントに集中できるようにします

小さなPythonスクリプトでうまくいくことができます。UIレンダリング部分を省略し(コードを確認して全体像を確認してください)、代わりにファイル解析ビットを表示します。

#!/usr/bin/env python3
"""
Aggregate Falco events to make it easier to override rules
Jose Vicente Nunez (kodegeek.com@protonmail.com)
"""
import json
import re
from argparse import ArgumentParser
from pathlib import Path
from rich.console import Console
from falcotutor.ui import EventDisplayApp, create_event_table, add_rows_to_create_event_table


def filter_events(journalctl_out: Path) -> dict[any, any]:
    """
    :param journalctl_out:
    :return:
    """
    with open(journalctl_out, 'r') as journalctl_file:
        for row in journalctl_file:
            if re.search("^{", row):
                data = json.loads(row)
                if 'rule' in data and 'output_fields' in data:
                    yield data


def aggregate_events(local_event: dict[any, any], aggregated_events: dict[any, any]):
    rule = local_event['rule']
    if rule not in aggregated_events:
        aggregated_events[rule] = {
            'count': 0,
            'priority': local_event['priority'],
            'last_timestamp': "",
            'last_fields': ""
        }
    aggregated_events[rule]['count'] += 1
    aggregated_events[rule]['last_timestamp'] = local_event['time']
    del local_event['output_fields']['evt.time']
    aggregated_events[rule]['last_fields'] = json.dumps(local_event['output_fields'], indent=True)


if __name__ == "__main__":
    CONSOLE = Console()
    AGGREGATED = {}
    PARSER = ArgumentParser(description=__doc__)
    PARSER.add_argument(
        "falco_event",
        action="store"
    )
    ARGS = PARSER.parse_args()
    try:
        event_table = create_event_table()
        for event in filter_events(ARGS.falco_event):
            aggregate_events(local_event=event, aggregated_events=AGGREGATED)
        add_rows_to_create_event_table(AGGREGATED, event_table)
        EventDisplayApp.run(
            event_file=ARGS.falco_event,
            title="Falco aggregated events report",
            event_table=event_table
        )
    except KeyboardInterrupt:
        CONSOLE.print("[bold]Program interrupted...[/bold]")

ファイルがディクショナリとしてロードされたら、イベントを集約するためにファイルを反復処理するだけで、結果をカウントでソートされたきちんとしたテーブルとして表示できます。

asciicast

ファルコルールの表示方法

あなたが私のようなら、あなたは常に/etc/falco/falco_rules.yamlファイルを見て、何が監視されているかを理解しています。これらのルールの簡単なビュー(コメント付きの詳細なYAMLファイルを見ずに)は素晴らしい追加です:

#!/usr/bin/env python3
"""
Show brief content of default Falco rule YAML files
Jose Vicente Nunez (kodegeek.com@protonmail.com)
"""
from argparse import ArgumentParser
from pathlib import Path
from rich.console import Console
import yaml
from falcotutor.ui import create_rules_table, add_rows_to_create_rules_table, RulesDisplayApp


def load_rulez(falco_rulez: Path) -> dict[any, any]:
    rulez = {}
    with open(falco_rulez, 'rt') as falco_file:
        for rule_data in yaml.full_load(falco_file):
            if 'rule' in rule_data:
                rule_name = rule_data['rule']
                del rule_data['rule']
                rulez[rule_name] = rule_data
    return rulez


if __name__ == "__main__":
    CONSOLE = Console()
    AGGREGATED = {}
    PARSER = ArgumentParser(description=__doc__)
    PARSER.add_argument(
        "falco_rules",
        action="store"
    )
    ARGS = PARSER.parse_args()
    try:
        RULES = load_rulez(ARGS.falco_rules)
        RULE_TBL = create_rules_table()
        add_rows_to_create_rules_table(lrules=RULES, rules_tbl=RULE_TBL)
        RulesDisplayApp.run(
            rules_file=ARGS.falco_rules,
            title="Falco brief rule display",
            rules_table=RULE_TBL
        )
    except KeyboardInterrupt:
        CONSOLE.print("[bold]Program interrupted...[/bold]")

たとえば、特定の基準(ルール名、優先度、有効/無効)によるルールフィルタリングを追加することで、このスクリプトを改善できます。このバージョンはフィルタリングを行いません。

asciicast

Falco統合

以前の実験から、おそらく2つのことに気づいたでしょう。

  1. イベントのペイロードにはホストがありません。問題のあるサーバーを特定する場合は、マルチホストイベントの報告方法を改善する必要があります(多くのホストからのjournalctlファイルの解析は実用的ではありません)。
  2. 一元化された場所でアラートを受け取りたい。私たちが釣りに行く代わりに、それらのイベントを「プッシュ」する方法があるといいでしょう。

これらのアラートを1か所に統合​​するときが来ました。

FalcoExporterの使用方法

Falcoエクスポーターを使用すると、FalcoアラートをPrometheusスクレーパーと共有できます。最初に/etc/falco/falco.yamlでgRPCを有効にする必要があります

# gRPC server using an unix socket
grpc:
  enabled: true
  bind_address: "unix:///var/run/falco.sock"
  # when threadiness is 0, Falco automatically guesses it depending on the number of online cores
  threadiness: 0

# gRPC output service.
# By default it is off.
# By enabling this all the output events will be kept in memory until you read them with a gRPC client.
# Make sure to have a consumer for them or leave this disabled.
grpc_output:
  enabled: true

Falcoを再起動します。

[root@macmini2 ~]# systemctl restart falco.service 
[root@macmini2 ~]# systemctl status falco.service 
● falco.service - Falco: Container Native Runtime Security
   Loaded: loaded (/usr/lib/systemd/system/falco.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-05-01 20:35:01 EDT; 26s ago
     Docs: https://falco.org/docs/
  Process: 28285 ExecStartPre=/sbin/modprobe falco (code=exited, status=0/SUCCESS)
 Main PID: 28288 (falco)
    Tasks: 11 (limit: 2310)
   Memory: 80.9M
   CGroup: /system.slice/falco.service
           └─28288 /usr/bin/falco --pidfile=/var/run/falco.pid

May 01 20:35:01 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 20:35:01 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 20:35:01 macmini2 falco[28288]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 20:35:01 macmini2 falco[28288]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 20:35:01 macmini2 falco[28288]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 20:35:02 macmini2 falco[28288]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 20:35:03 macmini2 falco[28288]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
May 01 20:35:04 macmini2 falco[28288]: Starting internal webserver, listening on port 8765
May 01 20:35:04 macmini2 falco[28288]: gRPC server threadiness equals to 2
May 01 20:35:04 macmini2 falco[28288]: Starting gRPC server at unix:///var/run/falco.sock

すべてが正常であることをすばやく確認します(リマインダー、Falcoエージェントはmacmini2で実行されています):

josevnz@raspberrypi:~$ curl --fail http://macmini2:8765/healthz
{"status": "ok"}josevnz@raspberrypi:~$

次に、falco-exporterを実行します。簡単にするために、コマンドラインでいくつかのオーバーライドを使用してDockerコンテナを使用します。

[root@macmini2 ~]# docker run --restart always --name falco-exporter --detach --volume /var/run/falco.sock:/var/run/falco.sock --network=host falcosecurity/falco-exporter --listen-address 192.168.1.16:9376
7d157af0251ea4bc73b8c355a74eaf4dd24a5348cbe3f5f2ea9d7147c6c366c8
[root@macmini2 ~]# docker logs falco-exporter
2022/05/02 00:56:30 connecting to gRPC server at unix:///var/run/falco.sock (timeout 2m0s)
2022/05/02 00:56:30 listening on http://192.168.1.16:9376/metrics
2022/05/02 00:56:30 connected to gRPC server, subscribing events stream
2022/05/02 00:56:30 ready

# Check with CURL if the URL is reachable
[root@macmini2 ~]# curl http://192.168.1.16:9376/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 18
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.14.15"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.011112e+06

完全を期すために、ノードエクスポーターを使用してホストパフォーマンスメトリックをキャプチャする方法も示します(後でFalcoが使用するリソースの数を監視し、インストールによってサーバーに悪影響が及ばないようにするために使用します)。

docker run --detach --net="host" --pid="host" --volume "/:/host:ro,rslave" quay.io/prometheus/node-exporter:latest --path.rootfs=/host

node-exporterとfalco-exporterは、データのスクレイプが必要なすべてのホストで実行されます。次に、これらすべてのメトリックを1つの場所に収集するのを待つ必要があります。そのためには、Prometheusエージェントを使用します。

---
# /etc/prometheus.yaml on raspberrypi
global:
    scrape_interval: 30s
    evaluation_interval: 30s
    scrape_timeout: 10s
    external_labels:
        monitor: 'nunez-family-monitor'

scrape_configs:
  - job_name: 'falco-exporter'
    static_configs:
      - targets: ['macmini2.home:9376']
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['macmini2.home:9100', 'raspberrypi.home:9100', 'dmaf5:9100']
  - job_name: 'docker-exporter'
    static_configs:
      - targets: ['macmini2.home:9323', 'raspberrypi.home:9323', 'dmaf5:9323']

    tls_config:
      insecure_skip_verify: true

次に、Prometheusスクレーパーが各ノードと通信できることを確認します。WebUIにアクセスします。

プロメテウス-ラズベリーパイ

いいですね、プロメテウスはファルコをこすり落とすことができます。簡単なクエリを実行して、いくつかのイベントを確認することもできます。

プロメテウス-クエリ-ファルコ

次に、イベントのUIビューを設定する必要があります。そのために、Grafanaを使用します。

Grafanaをインストールする方法はたくさんあります。私の場合、Grafana Dockerコンテナーを使用します(Prometheusが実行されているのと同じホストでGrafanaを実行します:raspberripi.home):

docker pull grafana/grafana:main-ubuntu
mkdir -p /data/grafana
chown syslog /data/grafana
docker run --user 104 --name grafana --detach --tty --volume /data/grafana:/var/lib/grafana -p 3000:3000 grafana/grafana:main-ubuntu

Grafanaが表示されたら、パスワードを変更する必要があります。また、Prometheusに接続する必要があります。

grafana-prometheus-falco

Grafanaが起動したら、ここで説明するようFalcoダッシュボードをインポートできます。

falco-grafana-統合

ダッシュボードがインポートされると、インストールされているホストでFalcoをトリガーするいくつかのイベントを生成できます。

[root@macmini2 ~]# for i in $(seq 1 60); do docker run --rm --interactive --tty --privileged fedora:latest /bin/bash -c ls; touch /root/test; rm -f /root/test; sleep 1; done

しばらくすると、Grafanaダッシュボードに次のようなものが表示されます。

grafana-falco-ダッシュボード

イベントは流れており、どのホストからイベントが来たかを確認できます。

Falcoイベントのアラートを作成する方法

理想的には、GrafanaでFalcoイベントがある場合は、これらの実用的なアイテムを作成して、そこからアラートを生成できます。

重要ではないアラートに襲われたくないので、最初に知っておくべきことは、フィルタリングするイベントのレベルです。

 

grafana-priority-events

優先度が3未満の場合は、アラートとして扱われます。

Grafanaにはアラートの設定方法に関する優れたドキュメントがあるため、ここでは最終結果のみを示します。

grafana-falco-alert-definition

次のステップは、アラートをどこかに送信することです。

アラートはどこかに行く必要がある–Discordを使用して連絡先を定義する方法

この例では、アラートの最後としてDiscordを使用します。Discordには、WebHookのセットアップ方法に関する非常に詳細なガイドがあるため、ここでは、DiscordWebhookの最終結果のみを示します。

grafana-discord-webhook

そのURLをコピーしてから、Discord Webhookを使用する新しいGrafanaコンタクトポイントを構成します(これをすべてのアラートのデフォルトのコンタクトポイントに設定しています)。

grafana-contact-point-discord

そこから、このパイプラインが機能することを確認するために、Discordにテストメッセージを送信できます。

grafana-discord-events-test

近づいています。ここまででアラート定義に戻ると、アラートが「発火」状態になっていることがわかります。

grafana-falco-alert-firing

そして、すべてがうまくいけば、Discordで最初のFalcoアラートも表示されます。

grafana-falco-discord-alert

ここで、journalctl出力で取得したすべてのフィールドを確認できます。違いは、これらのメッセージはすべて、Falco-Prometheus-Grafanaブリッジを定義するすべてのサーバーから送信されることです。

佳作:Falcon Sidekick /FalconSidekickを使用してアラートを集約する方法-UI

Falco Sidekickは、 Falco Sidekick-UIなど、イベントを収集して他の宛先に送信するもう1つの方法です。ただし、元のホストはわかりません(少なくともFalco 0.31.1まで)。

これは、K8sクラスターまたはコンテナー化されたアプリケーションからのアラートの問題ではない可能性が高く、イメージ名から多くの情報が得られます。ただし、イベントがベアメタル環境で発生し、2台以上のマシンがある場合、それは頭痛の種になります。

そのため、ここではSidekickについては説明しません。当面は、Grafanaの統合に固執することをお勧めします。

  ソース:https ://www.freecodecamp.org/news/secure-server-infrastructure-clouds-using-falco-prometheus-grafana-and-docker/

 #falco  #grafana #docker #cloud 

What is GEEK

Buddha Community

Falco、Prometheus、Grafana、Dockerを使用してサーバーインフラストラクチャクラウドを保護する

Falco、Prometheus、Grafana、Dockerを使用してサーバーインフラストラクチャクラウドを保護する

私は最近、コンテナとアプリケーションを機能させ続ける方法を探していました。具体的には、構成の異常を検出することに興味がありました。少し調べた後、私はFalcoに出くわしました。

私が見つけたのは、多くの機能と優れたドキュメントを備えた非常に完全なオープンソースプラットフォームでした。だから私はあなたと私の経験を共有したいと思いました。

この記事では何を取り上げますか?

  • イベント(異常/違反)を監視するホストにFalcoエージェントをインストールする方法
  • Falcoを調整して誤検知を減らし、本当に必要な情報を取得する方法
  • Prometheusを使用して、輸出業者とスクレーパーの助けを借りて、Falcoイベントを中央の場所に収集する方法。
  • 最後に、視覚化とアラートのためにスクレーパーをGrafanaに接続する方法

このチュートリアルには何が必要ですか?

  • Linuxがインストールされている1つまたは複数のマシン。仮想マシンが機能するはずです。
  • Docker、RPM、およびsystemdプロセスをインストール/セットアップできるようにするには、スーパーユーザー権限が必要です。
  • Dockerコンテナーを使用するため、Dockerの基本的な知識が必要です。
  • 構成をテストおよび改善するためのスクリプトをいくつか作成するため、Python/Bashの実用的な知識。

最後に、次の各コンポーネントをセットアップできるようになります。

falco_monitoring

恐れることはありません。ドキュメントへのリンクと、これらの各タスクの詳細な説明を提供します。

目次

  1. ファルコとは?
  2. Falcoのインストール方法
  3. 基本構成
  4. デフォルト設定をテストする方法
  5. デフォルトは必ずしも良いとは限りません
  6. Falco統合
  7. 詳細はこちら

ファルコとは?

このツールを説明する最良の方法は、ツールで何ができるかを学ぶことです

Falcoは、Linuxシステムコールの実行に関連するすべての動作を検出して警告することができます。

Falcoアラートは、特定のシステムコールとその引数の使用、および呼び出しプロセスのプロパティによってトリガーできます。たとえば、Falcoは、以下を含むがこれらに限定されないインシデントを簡単に検出できます。

  • Kubernetesのコンテナまたはポッド内でシェルが実行されています。
  • コンテナが特権モードで実行されているか、ホストから/procなどの機密パスをマウントしています。
  • サーバープロセスは、予期しないタイプの子プロセスを生成しています。
  • / etc/shadowなどの機密ファイルの予期しない読み取り。
  • デバイス以外のファイルが/devに書き込まれます。
  • lsなどの標準のシステムバイナリは、アウトバウンドネットワーク接続を確立しています。
  • 特権ポッドがKubernetesクラスターで開始されます。

Falcoのインストール方法

RPMを使用してFalcoをインストールします(apt-get、さらにはDockerコンテナーにも同様の手順があります)。私の場合、ネイティブインストールが最適であり、RPMによって非常に簡単に実行できると感じました。

[josevnz@macmini2 ~]$ sudo -i dnf install https://download.falco.org/packages/rpm/falco-0.31.1-x86_64.rpm
Last metadata expiration check: 2:53:53 ago on Sun 01 May 2022 04:13:09 PM EDT.
falco-0.31.1-x86_64.rpm                                                                                                                                                                                                       1.7 MB/s |  12 MB     00:07    
Dependencies resolved.
==============================================================================================================================================================================================================================================================
 Package                                                          Architecture                                      Version                                                                     Repository                                               Size
==============================================================================================================================================================================================================================================================
Installing:
 falco                                                            x86_64                                            0.31.1-1                                                                    @commandline                                             12 M
Installing dependencies:
 dkms                                                             noarch                                            2.8.1-4.20200214git5ca628c.fc30                                             updates                                                  78 k
 elfutils-libelf-devel                                            x86_64                                            0.179-2.fc30                                                                updates                                                  27 k
 kernel-devel                                                     x86_64                                            5.6.13-100.fc30                                                             updates                                                  14 M

Transaction Summary
==============================================================================================================================================================================================================================================================
Install  4 Packages

Total size: 26 M
Total download size: 14 M
Installed size: 92 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): elfutils-libelf-devel-0.179-2.fc30.x86_64.rpm                                                                                                                                                                          253 kB/s |  27 kB     00:00    
(2/3): dkms-2.8.1-4.20200214git5ca628c.fc30.noarch.rpm                                                                                                                                                                        342 kB/s |  78 kB     00:00    
(3/3): kernel-devel-5.6.13-100.fc30.x86_64.rpm                                                                                                                                                                                1.9 MB/s |  14 MB     00:07    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                         1.8 MB/s |  14 MB     00:07     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                      1/1 
  Installing       : kernel-devel-5.6.13-100.fc30.x86_64                                                                                                                                                                                                  1/4 
  Running scriptlet: kernel-devel-5.6.13-100.fc30.x86_64                                                                                                                                                                                                  1/4 
  Installing       : elfutils-libelf-devel-0.179-2.fc30.x86_64                                                                                                                                                                                            2/4 
  Installing       : dkms-2.8.1-4.20200214git5ca628c.fc30.noarch                                                                                                                                                                                          3/4 
  Running scriptlet: dkms-2.8.1-4.20200214git5ca628c.fc30.noarch                                                                                                                                                                                          3/4 
  Running scriptlet: falco-0.31.1-1.x86_64                                                                                                                                                                                                                4/4 
  Installing       : falco-0.31.1-1.x86_64                                                                                                                                                                                                                4/4 
  Running scriptlet: falco-0.31.1-1.x86_64                                  

基本構成

非常に基本的な出力処理を行う場合を除いて、JSON出力を有効にします。

# Whether to output events in json or text
json_output: true

理由はすぐに明らかになります。

次に、Falcoエージェントを開始します。

[josevnz@macmini2 falco]$ sudo systemctl start falco.service 
[josevnz@macmini2 falco]$ sudo systemctl status falco.service 
● falco.service - Falco: Container Native Runtime Security
   Loaded: loaded (/usr/lib/systemd/system/falco.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-05-01 19:20:52 EDT; 1s ago
     Docs: https://falco.org/docs/
  Process: 26887 ExecStartPre=/sbin/modprobe falco (code=exited, status=0/SUCCESS)
 Main PID: 26888 (falco)
    Tasks: 1 (limit: 2310)
   Memory: 65.8M
   CGroup: /system.slice/falco.service
           └─26888 /usr/bin/falco --pidfile=/var/run/falco.pid

May 01 19:20:52 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 19:20:52 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 19:20:52 macmini2 falco[26888]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 19:20:52 macmini2 falco[26888]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 19:20:52 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 19:20:53 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 19:20:54 macmini2 falco[26888]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:

デフォルト設定をテストする方法

構成によっては、Falcoを起動した直後にイベントが発生する場合と発生しない場合があります。

[josevnz@macmini2 falco]$ sudo journalctl --unit falco --follow
-- Logs begin at Tue 2021-05-25 00:15:22 EDT. --
May 01 19:20:52 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 19:20:52 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 19:20:52 macmini2 falco[26888]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 19:20:52 macmini2 falco[26888]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 19:20:52 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 19:20:53 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 19:20:54 macmini2 falco[26888]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
May 01 19:20:55 macmini2 falco[26888]: Starting internal webserver, listening on port 8765

心配しないでください。Falcoに警告とアラートを記録させるいくつかのコマンドを実行します。これがどのように機能するかを見てみましょう!

特権コンテナを実行する方法

特権コンテナの使用は悪い習慣と見なされているので、このイベントがFalcoによって検出されるかどうかを見てみましょう。

[josevnz@macmini2 ~]$ docker run --rm --interactive --tty --privileged --volume /etc/shadow:/mnt/shadow fedora:latest ls -l /mnt/shadow
----------. 1 root root 1198 Nov 21 20:51 /mnt/shadow

そして私たちのファルコログ?

May 01 19:29:32 macmini2 falco[26888]: {"output":"19:29:32.918828894: Informational Privileged container started (user=root user_loginuid=0 command=container:bfb9637a47a6 kind_lumiere (id=bfb9637a47a6) image=fedora:latest)","priority":"Informational","rule":"Launch Privileged Container","source":"syscall","tags":["cis","container","mitre_lateral_movement","mitre_privilege_escalation"],"time":"2022-05-01T23:29:32.918828894Z", "output_fields": {"container.id":"bfb9637a47a6","container.image.repository":"fedora","container.image.tag":"latest","container.name":"kind_lumiere","evt.time":1651447772918828894,"proc.cmdline":"container:bfb9637a47a6","user.loginuid":0,"user.name":"root"}}

情報イベントとして表示されます。間違いなく注目すべきものの1つです。コンテナ上のアプリケーションに昇格された特権が必要かどうかを自問してください。

また、各メッセージにタグが付いていることにお気づきかもしれません。「mitre_*」のものに注意してください。これらは、攻撃と軽減に関するMitreAttackナレッジベースに関連しています。うん、あなたはそれらを読むのに少し時間を費やすでしょう。

/rootディレクトリにファイルを作成する方法

この例は、コンテナ内のボリュームと組み合わせたrootユーザーを悪用する方法を示しています...

[josevnz@macmini2 ~]$ docker run --rm --interactive --tty --user root --volume /root:/mnt/ fedora:latest touch /mnt/test_file
[josevnz@macmini2 ~]$ 

ファルコ反応:

May 01 19:32:02 macmini2 falco[26888]: {"output":"19:32:02.434286167: Informational Container with sensitive mount started (user=root user_loginuid=0 command=container:ef061174c7ef distracted_lalande (id=ef061174c7ef) image=fedora:latest mounts=/root:/mnt::true:rprivate)","priority":"Informational","rule":"Launch Sensitive Mount Container","source":"syscall","tags":["cis","container","mitre_lateral_movement"],"time":"2022-05-01T23:32:02.434286167Z", "output_fields": {"container.id":"ef061174c7ef","container.image.repository":"fedora","container.image.tag":"latest","container.mounts":"/root:/mnt::true:rprivate","container.name":"distracted_lalande","evt.time":1651447922434286167,"proc.cmdline":"container:ef061174c7ef","user.loginuid":0,"user.name":"root"}}

敏感なマウントが検出されました!

/binにファイルを作成して賭け金を上げましょう

さて、これを行うとしましょう:

[josevnz@macmini2 ~]$ sudo -i
[root@macmini2 ~]# touch /bin/should_not_be_here

ファルコはそれについてどう思いますか?

May 01 19:36:41 macmini2 falco[26888]: {"output":"19:36:41.237634398: Error File below a known binary directory opened for writing (user=root user_loginuid=1000 command=touch /bin/should_not_be_here file=/bin/should_not_be_here parent=bash pcmdline=bash gparent=sudo container_id=host image=<NA>)","priority":"Error","rule":"Write below binary dir","source":"syscall","tags":["filesystem","mitre_persistence"],"time":"2022-05-01T23:36:41.237634398Z", "output_fields": {"container.id":"host","container.image.repository":null,"evt.time":1651448201237634398,"fd.name":"/bin/should_not_be_here","proc.aname[2]":"sudo","proc.cmdline":"touch /bin/should_not_be_here","proc.pcmdline":"bash","proc.pname":"bash","user.loginuid":1000,"user.name":"root"}}

エラー、書き込み用に開かれたバイナリディレクトリ。良いキャッチ。

デフォルトは必ずしも良いとは限りません

Falcoがしばらく実行された後、どのような種類のイベントを無視し、どのイベントを調査したいのかを把握することをお勧めします。

最初のステップは、ペイロードでJSON形式を使用して、すべてのイベントのリストを取得することです。

sudo journalctl --unit falco --no-page --output=cat > /tmp/falco_json_lines.txt

'output = cat'は、タイムスタンプなしでメッセージペイロードを提供するようにjournalctlに指示します(心配しないでください。JSONメッセージ自体にタイムスタンプがあります)。

Starting Falco: Container Native Runtime Security...
Started Falco: Container Native Runtime Security.
Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
Falco initialized with configuration file /etc/falco/falco.yaml
Loading rules from file /etc/falco/falco_rules.yaml:
Loading rules from file /etc/falco/falco_rules.local.yaml:
Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Starting internal webserver, listening on port 8765
{"output":"19:29:32.918828894: Informational Privileged container started (user=root user_loginuid=0 command=container:bfb9637a47a6 kind_lumiere (id=bfb9637a47a6) image=fedora:latest)","priority":"Informational","rule":"Launch Privileged Container","source":"syscall","tags":["cis","container","mitre_lateral_movement","mitre_privilege_escalation"],"time":"2022-05-01T23:29:32.918828894Z", "output_fields": {"container.id":"bfb9637a47a6","container.image.repository":"fedora","container.image.tag":"latest","container.name":"kind_lumiere","evt.time":1651447772918828894,"proc.cmdline":"container:bfb9637a47a6","user.loginuid":0,"user.name":"root"}}
{"output":"19:32:02.434286167: Informational Container with sensitive mount started (user=root user_loginuid=0 command=container:ef061174c7ef distracted_lalande (id=ef061174c7ef) image=fedora:latest mounts=/root:/mnt::true:rprivate)","priority":"Informational","rule":"Launch Sensitive Mount Container","source":"syscall","tags":["cis","container","mitre_lateral_movement"],"time":"2022-05-01T23:32:02.434286167Z", "output_fields": {"container.id":"ef061174c7ef","container.image.repository":"fedora","container.image.tag":"latest","container.mounts":"/root:/mnt::true:rprivate","container.name":"distracted_lalande","evt.time":1651447922434286167,"proc.cmdline":"container:ef061174c7ef","user.loginuid":0,"user.name":"root"}}

これまでのところ面白そうですが、これはどうですか?

{"output":"23:04:10.609949471: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=openat command=bash fd.name=/home/josevnz/.bash_history-01112.tmp name=/home/josevnz/.bash_history-01112.tmp path=<NA> oldpath=<NA> host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.609949471Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":"/home/josevnz/.bash_history-01112.tmp","evt.arg.oldpath":null,"evt.arg.path":null,"evt.time":1651633450609949471,"evt.type":"openat","fd.name":"/home/josevnz/.bash_history-01112.tmp","proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.635602857: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=openat command=bash fd.name=/home/josevnz/.bash_history-01627.tmp name=/home/josevnz/.bash_history-01627.tmp path=<NA> oldpath=<NA> host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.635602857Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":"/home/josevnz/.bash_history-01627.tmp","evt.arg.oldpath":null,"evt.arg.path":null,"evt.time":1651633450635602857,"evt.type":"openat","fd.name":"/home/josevnz/.bash_history-01627.tmp","proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.635851215: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=rename command=bash fd.name=<NA> name=<NA> path=<NA> oldpath=/home/josevnz/.bash_history-01627.tmp host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.635851215Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":null,"evt.arg.oldpath":"/home/josevnz/.bash_history-01627.tmp","evt.arg.path":null,"evt.time":1651633450635851215,"evt.type":"rename","fd.name":null,"proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.661829867: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=rename command=bash fd.name=<NA> name=<NA> path=<NA> oldpath=/home/josevnz/.bash_history-01112.tmp host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.661829867Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":null,"evt.arg.oldpath":"/home/josevnz/.bash_history-01112.tmp","evt.arg.path":null,"evt.time":1651633450661829867,"evt.type":"rename","fd.name":null,"proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}

これは通常の/正当な操作です。このルールを強化するか、完全に削除する方法を見つけましょう。

まず、/etc/falco/falco_rules.yamlファイルを開き、ルール「シェル履歴の削除または名前変更」(前に見たJSON出力)を探します。

- list: docker_binaries
  items: [docker, dockerd, exe, docker-compose, docker-entrypoi, docker-runc-cur, docker-current, dockerd-current]

 macro: var_lib_docker_filepath
  condition: (evt.arg.name startswith /var/lib/docker or fd.name startswith /var/lib/docker)

- rule: Delete or rename shell history
  desc: Detect shell history deletion
  condition: >
    (modify_shell_history or truncate_shell_history) and
       not var_lib_docker_filepath and
       not proc.name in (docker_binaries)
  output: >
    Shell history had been deleted or renamed (user=%user.name user_loginuid=%user.loginuid type=%evt.type command=%proc.cmdline fd.name=%fd.name name=%evt.arg.name path=%evt.arg.path oldpath=%evt.arg.oldpath %container.info)
  priority:
    WARNING
  tags: [process, mitre_defense_evasion]

Falcoのルールは、公式ドキュメントで詳細に説明されています。この作品を見るだけで、いくつかのことに気付くでしょう。

条件について:

  1. 複雑なロジックをサポートし、
  2. のようなマクロvar_lib_docker_filepath
  3. のようなリスト(docker_binaries)
  4. のようなフィールドを持つ特別な変数proc.name

このファイルは変更しないことをお勧めします。/etc/falco/falco_rules.local.yaml代わりに、 :で必要なものをオーバーライドする必要があります。

# Add new rules, like this one
# - rule: The program "sudo" is run in a container
#   desc: An event will trigger every time you run sudo in a container
#   condition: evt.type = execve and evt.dir=< and container.id != host and proc.name = sudo
#   output: "Sudo run in container (user=%user.name %container.info parent=%proc.pname cmdline=%proc.cmdline)"
#   priority: ERROR
#   tags: [users, container]

# Or override/append to any rule, macro, or list from the Default Rules

例として、スーパーユーザー(root)の履歴が上書きされる場合は注意しますが、それ以外の場合は問題ないとします。最良の部分は、ルール全体をオーバーライドする必要がないことです。

したがって、元のルールには条件が追加されます。

- rule: Delete or rename shell history
  append: true
  condition: and user.name=root

ルールが適切に記述されていることを検証することは常に良い考えです。そのためには、元のルールとオーバーライドを一緒にチェックするようにFalcoに指示する必要があります。

[root@macmini2 ~]# falco --validate /etc/falco/falco_rules.yaml --validate /etc/falco/falco_rules.local.yaml 
Fri May  6 20:48:00 2022: Validating rules file(s):
Fri May  6 20:48:00 2022:    /etc/falco/falco_rules.yaml
Fri May  6 20:48:00 2022:    /etc/falco/falco_rules.local.yaml
/etc/falco/falco_rules.yaml: Ok
/etc/falco/falco_rules.local.yaml: Ok
Fri May  6 20:48:01 2022: Ok

# If the rules are OK, restart Falco
[root@macmini2 ~]# systemctl restart falco.service

Pythonで簡単なイベントエクスプローラーを作成する方法

どのルールがノイズであり、どのルールが有用であるかを理解するのは面倒であることにおそらく同意するでしょう。

このデータを正規化する必要があり、次のようなPythonスクリプトを使用します。

  • 非JSONデータを削除する
  • タイムスタンプなしでイベントタイプを集約します
  • いくつかの集計統計を生成して、システムで最も頻繁に発生するイベントに集中できるようにします

小さなPythonスクリプトでうまくいくことができます。UIレンダリング部分を省略し(コードを確認して全体像を確認してください)、代わりにファイル解析ビットを表示します。

#!/usr/bin/env python3
"""
Aggregate Falco events to make it easier to override rules
Jose Vicente Nunez (kodegeek.com@protonmail.com)
"""
import json
import re
from argparse import ArgumentParser
from pathlib import Path
from rich.console import Console
from falcotutor.ui import EventDisplayApp, create_event_table, add_rows_to_create_event_table


def filter_events(journalctl_out: Path) -> dict[any, any]:
    """
    :param journalctl_out:
    :return:
    """
    with open(journalctl_out, 'r') as journalctl_file:
        for row in journalctl_file:
            if re.search("^{", row):
                data = json.loads(row)
                if 'rule' in data and 'output_fields' in data:
                    yield data


def aggregate_events(local_event: dict[any, any], aggregated_events: dict[any, any]):
    rule = local_event['rule']
    if rule not in aggregated_events:
        aggregated_events[rule] = {
            'count': 0,
            'priority': local_event['priority'],
            'last_timestamp': "",
            'last_fields': ""
        }
    aggregated_events[rule]['count'] += 1
    aggregated_events[rule]['last_timestamp'] = local_event['time']
    del local_event['output_fields']['evt.time']
    aggregated_events[rule]['last_fields'] = json.dumps(local_event['output_fields'], indent=True)


if __name__ == "__main__":
    CONSOLE = Console()
    AGGREGATED = {}
    PARSER = ArgumentParser(description=__doc__)
    PARSER.add_argument(
        "falco_event",
        action="store"
    )
    ARGS = PARSER.parse_args()
    try:
        event_table = create_event_table()
        for event in filter_events(ARGS.falco_event):
            aggregate_events(local_event=event, aggregated_events=AGGREGATED)
        add_rows_to_create_event_table(AGGREGATED, event_table)
        EventDisplayApp.run(
            event_file=ARGS.falco_event,
            title="Falco aggregated events report",
            event_table=event_table
        )
    except KeyboardInterrupt:
        CONSOLE.print("[bold]Program interrupted...[/bold]")

ファイルがディクショナリとしてロードされたら、イベントを集約するためにファイルを反復処理するだけで、結果をカウントでソートされたきちんとしたテーブルとして表示できます。

asciicast

ファルコルールの表示方法

あなたが私のようなら、あなたは常に/etc/falco/falco_rules.yamlファイルを見て、何が監視されているかを理解しています。これらのルールの簡単なビュー(コメント付きの詳細なYAMLファイルを見ずに)は素晴らしい追加です:

#!/usr/bin/env python3
"""
Show brief content of default Falco rule YAML files
Jose Vicente Nunez (kodegeek.com@protonmail.com)
"""
from argparse import ArgumentParser
from pathlib import Path
from rich.console import Console
import yaml
from falcotutor.ui import create_rules_table, add_rows_to_create_rules_table, RulesDisplayApp


def load_rulez(falco_rulez: Path) -> dict[any, any]:
    rulez = {}
    with open(falco_rulez, 'rt') as falco_file:
        for rule_data in yaml.full_load(falco_file):
            if 'rule' in rule_data:
                rule_name = rule_data['rule']
                del rule_data['rule']
                rulez[rule_name] = rule_data
    return rulez


if __name__ == "__main__":
    CONSOLE = Console()
    AGGREGATED = {}
    PARSER = ArgumentParser(description=__doc__)
    PARSER.add_argument(
        "falco_rules",
        action="store"
    )
    ARGS = PARSER.parse_args()
    try:
        RULES = load_rulez(ARGS.falco_rules)
        RULE_TBL = create_rules_table()
        add_rows_to_create_rules_table(lrules=RULES, rules_tbl=RULE_TBL)
        RulesDisplayApp.run(
            rules_file=ARGS.falco_rules,
            title="Falco brief rule display",
            rules_table=RULE_TBL
        )
    except KeyboardInterrupt:
        CONSOLE.print("[bold]Program interrupted...[/bold]")

たとえば、特定の基準(ルール名、優先度、有効/無効)によるルールフィルタリングを追加することで、このスクリプトを改善できます。このバージョンはフィルタリングを行いません。

asciicast

Falco統合

以前の実験から、おそらく2つのことに気づいたでしょう。

  1. イベントのペイロードにはホストがありません。問題のあるサーバーを特定する場合は、マルチホストイベントの報告方法を改善する必要があります(多くのホストからのjournalctlファイルの解析は実用的ではありません)。
  2. 一元化された場所でアラートを受け取りたい。私たちが釣りに行く代わりに、それらのイベントを「プッシュ」する方法があるといいでしょう。

これらのアラートを1か所に統合​​するときが来ました。

FalcoExporterの使用方法

Falcoエクスポーターを使用すると、FalcoアラートをPrometheusスクレーパーと共有できます。最初に/etc/falco/falco.yamlでgRPCを有効にする必要があります

# gRPC server using an unix socket
grpc:
  enabled: true
  bind_address: "unix:///var/run/falco.sock"
  # when threadiness is 0, Falco automatically guesses it depending on the number of online cores
  threadiness: 0

# gRPC output service.
# By default it is off.
# By enabling this all the output events will be kept in memory until you read them with a gRPC client.
# Make sure to have a consumer for them or leave this disabled.
grpc_output:
  enabled: true

Falcoを再起動します。

[root@macmini2 ~]# systemctl restart falco.service 
[root@macmini2 ~]# systemctl status falco.service 
● falco.service - Falco: Container Native Runtime Security
   Loaded: loaded (/usr/lib/systemd/system/falco.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-05-01 20:35:01 EDT; 26s ago
     Docs: https://falco.org/docs/
  Process: 28285 ExecStartPre=/sbin/modprobe falco (code=exited, status=0/SUCCESS)
 Main PID: 28288 (falco)
    Tasks: 11 (limit: 2310)
   Memory: 80.9M
   CGroup: /system.slice/falco.service
           └─28288 /usr/bin/falco --pidfile=/var/run/falco.pid

May 01 20:35:01 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 20:35:01 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 20:35:01 macmini2 falco[28288]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 20:35:01 macmini2 falco[28288]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 20:35:01 macmini2 falco[28288]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 20:35:02 macmini2 falco[28288]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 20:35:03 macmini2 falco[28288]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
May 01 20:35:04 macmini2 falco[28288]: Starting internal webserver, listening on port 8765
May 01 20:35:04 macmini2 falco[28288]: gRPC server threadiness equals to 2
May 01 20:35:04 macmini2 falco[28288]: Starting gRPC server at unix:///var/run/falco.sock

すべてが正常であることをすばやく確認します(リマインダー、Falcoエージェントはmacmini2で実行されています):

josevnz@raspberrypi:~$ curl --fail http://macmini2:8765/healthz
{"status": "ok"}josevnz@raspberrypi:~$

次に、falco-exporterを実行します。簡単にするために、コマンドラインでいくつかのオーバーライドを使用してDockerコンテナを使用します。

[root@macmini2 ~]# docker run --restart always --name falco-exporter --detach --volume /var/run/falco.sock:/var/run/falco.sock --network=host falcosecurity/falco-exporter --listen-address 192.168.1.16:9376
7d157af0251ea4bc73b8c355a74eaf4dd24a5348cbe3f5f2ea9d7147c6c366c8
[root@macmini2 ~]# docker logs falco-exporter
2022/05/02 00:56:30 connecting to gRPC server at unix:///var/run/falco.sock (timeout 2m0s)
2022/05/02 00:56:30 listening on http://192.168.1.16:9376/metrics
2022/05/02 00:56:30 connected to gRPC server, subscribing events stream
2022/05/02 00:56:30 ready

# Check with CURL if the URL is reachable
[root@macmini2 ~]# curl http://192.168.1.16:9376/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 18
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.14.15"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.011112e+06

完全を期すために、ノードエクスポーターを使用してホストパフォーマンスメトリックをキャプチャする方法も示します(後でFalcoが使用するリソースの数を監視し、インストールによってサーバーに悪影響が及ばないようにするために使用します)。

docker run --detach --net="host" --pid="host" --volume "/:/host:ro,rslave" quay.io/prometheus/node-exporter:latest --path.rootfs=/host

node-exporterとfalco-exporterは、データのスクレイプが必要なすべてのホストで実行されます。次に、これらすべてのメトリックを1つの場所に収集するのを待つ必要があります。そのためには、Prometheusエージェントを使用します。

---
# /etc/prometheus.yaml on raspberrypi
global:
    scrape_interval: 30s
    evaluation_interval: 30s
    scrape_timeout: 10s
    external_labels:
        monitor: 'nunez-family-monitor'

scrape_configs:
  - job_name: 'falco-exporter'
    static_configs:
      - targets: ['macmini2.home:9376']
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['macmini2.home:9100', 'raspberrypi.home:9100', 'dmaf5:9100']
  - job_name: 'docker-exporter'
    static_configs:
      - targets: ['macmini2.home:9323', 'raspberrypi.home:9323', 'dmaf5:9323']

    tls_config:
      insecure_skip_verify: true

次に、Prometheusスクレーパーが各ノードと通信できることを確認します。WebUIにアクセスします。

プロメテウス-ラズベリーパイ

いいですね、プロメテウスはファルコをこすり落とすことができます。簡単なクエリを実行して、いくつかのイベントを確認することもできます。

プロメテウス-クエリ-ファルコ

次に、イベントのUIビューを設定する必要があります。そのために、Grafanaを使用します。

Grafanaをインストールする方法はたくさんあります。私の場合、Grafana Dockerコンテナーを使用します(Prometheusが実行されているのと同じホストでGrafanaを実行します:raspberripi.home):

docker pull grafana/grafana:main-ubuntu
mkdir -p /data/grafana
chown syslog /data/grafana
docker run --user 104 --name grafana --detach --tty --volume /data/grafana:/var/lib/grafana -p 3000:3000 grafana/grafana:main-ubuntu

Grafanaが表示されたら、パスワードを変更する必要があります。また、Prometheusに接続する必要があります。

grafana-prometheus-falco

Grafanaが起動したら、ここで説明するようFalcoダッシュボードをインポートできます。

falco-grafana-統合

ダッシュボードがインポートされると、インストールされているホストでFalcoをトリガーするいくつかのイベントを生成できます。

[root@macmini2 ~]# for i in $(seq 1 60); do docker run --rm --interactive --tty --privileged fedora:latest /bin/bash -c ls; touch /root/test; rm -f /root/test; sleep 1; done

しばらくすると、Grafanaダッシュボードに次のようなものが表示されます。

grafana-falco-ダッシュボード

イベントは流れており、どのホストからイベントが来たかを確認できます。

Falcoイベントのアラートを作成する方法

理想的には、GrafanaでFalcoイベントがある場合は、これらの実用的なアイテムを作成して、そこからアラートを生成できます。

重要ではないアラートに襲われたくないので、最初に知っておくべきことは、フィルタリングするイベントのレベルです。

 

grafana-priority-events

優先度が3未満の場合は、アラートとして扱われます。

Grafanaにはアラートの設定方法に関する優れたドキュメントがあるため、ここでは最終結果のみを示します。

grafana-falco-alert-definition

次のステップは、アラートをどこかに送信することです。

アラートはどこかに行く必要がある–Discordを使用して連絡先を定義する方法

この例では、アラートの最後としてDiscordを使用します。Discordには、WebHookのセットアップ方法に関する非常に詳細なガイドがあるため、ここでは、DiscordWebhookの最終結果のみを示します。

grafana-discord-webhook

そのURLをコピーしてから、Discord Webhookを使用する新しいGrafanaコンタクトポイントを構成します(これをすべてのアラートのデフォルトのコンタクトポイントに設定しています)。

grafana-contact-point-discord

そこから、このパイプラインが機能することを確認するために、Discordにテストメッセージを送信できます。

grafana-discord-events-test

近づいています。ここまででアラート定義に戻ると、アラートが「発火」状態になっていることがわかります。

grafana-falco-alert-firing

そして、すべてがうまくいけば、Discordで最初のFalcoアラートも表示されます。

grafana-falco-discord-alert

ここで、journalctl出力で取得したすべてのフィールドを確認できます。違いは、これらのメッセージはすべて、Falco-Prometheus-Grafanaブリッジを定義するすべてのサーバーから送信されることです。

佳作:Falcon Sidekick /FalconSidekickを使用してアラートを集約する方法-UI

Falco Sidekickは、 Falco Sidekick-UIなど、イベントを収集して他の宛先に送信するもう1つの方法です。ただし、元のホストはわかりません(少なくともFalco 0.31.1まで)。

これは、K8sクラスターまたはコンテナー化されたアプリケーションからのアラートの問題ではない可能性が高く、イメージ名から多くの情報が得られます。ただし、イベントがベアメタル環境で発生し、2台以上のマシンがある場合、それは頭痛の種になります。

そのため、ここではSidekickについては説明しません。当面は、Grafanaの統合に固執することをお勧めします。

  ソース:https ://www.freecodecamp.org/news/secure-server-infrastructure-clouds-using-falco-prometheus-grafana-and-docker/

 #falco  #grafana #docker #cloud 

Monitoring YugabyteDB in Kubernetes with the Prometheus Operator and Grafana

This blog post is written by guest author Bhavin Gandhi, Software Engineer at InfraCloud Technologies, Inc. InfraCloud helps startups, SMBs, and enterprises scale by adopting cloud-native technologies.

Using the Prometheus Operator has become a common choice when it comes to running Prometheus in a Kubernetes cluster. It can manage Prometheus and Alertmanager for us with the help of CRDs in Kubernetes. The kube-prometheus-stack Helm chart (formerly known as prometheus-operator) comes with Grafana, node_exporter, and more out of the box.

In a previous blog post about Prometheus, we took a look at setting up Prometheus and Grafana using manifest files. We also explored a few of the metrics exposed by YugabyteDB. In this post, we will be setting up Prometheus and Grafana using the kube-prometheus-stack chart. And we will configure Prometheus to scrape YugabyteDB pods. At the end we will take a look at the YugabyteDB Grafana dashboard that can be used to visualize all the collected metrics.

Before we begin

Before we get started with the setup, make sure you have a Kubernetes 1.16+ cluster with kubectl pointing to it. You can create a GKE cluster or an equivalent in other cloud providers or use Minikube to create a local cluster.

We will be using Helm 3 to install the charts, make sure you have it installed on your machine as well.

Installing the Prometheus Operator

The kube-prometheus-stack Helm chart installs the required CRDs as well as the operator itself. The chart creates instances of Prometheus and Alertmanager, which are the custom resources managed by the operator.

It also comes with a set of components which one would expect to have while setting up monitoring in a Kubernetes cluster. These components include: node_exporter for collecting node level metrics, kube-state-metrics which exposes Kubernetes related metrics, and Grafana for visualizing the metrics collected by Prometheus. The chart also comes with default Grafana dashboards and Prometheus alerts.

To setup Helm with the prometheus-community and yugabytedb repositories, run the following command:

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories

$ helm repo add yugabytedb https://charts.yugabyte.com
"yugabytedb" has been added to your repositories

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "yugabytedb" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈ Happy Helming!⎈ 

To create a values file for kube-prometheus-stack, execute:

cat <<EOF > kube-prom-stack-values.yaml
grafana:
  dashboardProviders:
    dashboardproviders.yaml:
      apiVersion: 1
      providers:
      - name: 'default'
        orgId: 1
        folder: ''
        type: file
        disableDeletion: false
        editable: true
        options:
          path: /var/lib/grafana/dashboards/default

  dashboards:
    default:
      yugabytedb:
        url: https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/cloud/grafana/YugabyteDB.json
EOF

The above kube-prom-stack-values.yaml file tells Grafana to import the YugabyteDB dashboard from the given GitHub URL.

Now we will create a namespace and install kube-prometheus-stack in it.

$ kubectl create namespace monitoring
namespace/monitoring created

$ helm install prom prometheus-community/kube-prometheus-stack \
    --namespace monitoring \
    --values kube-prom-stack-values.yaml

To check if all the components are running fine, check the status of pods from the monitoring namespace.

$ kubectl get pods --namespace monitoring

NOTE: To keep things simple, we are not providing any other options to the Helm chart. Make sure you go through the README of kube-prometheus-stack and grafana chart to explore other options that you may need.

Installing and scraping YugabyteDB

With the monitoring setup done, let’s install the YugabyteDB Helm chart. It will install YugabyteDB and configure Prometheus to scrape (collect) the metrics from our pods. It uses the ServiceMonitor Custom Resource (CR) to achieve that.

To create a namespace and install the chart, run the following command:

$ kubectl create namespace yb-demo
namespace/yb-demo created

$ helm install cluster-1 yugabytedb/yugabyte \
    --namespace yb-demo \
    --set serviceMonitor.enabled=true \
    --set serviceMonitor.extraLabels.release=prom

If you are using Minikube, refer to this documentation section and add the resource requirements accordingly.

To check the status of all the pods from the yb-demo namespace, execute the following command. Wait till all the pods have a Running status.

#databases #distributed sql #kubernetes #grafana #prometheus #prometheus operator

How to Secure Server Infrastructure Clouds using Falco, Prometheus, Grafana and Docker

I was recently looking for a way to keep tabs on our containers and applications at work. Specifically, I was interested in detecting anomalies in the configuration. After a little research, I stumbled on Falco.

What I found was a very complete Open Source platform with lots of features and excellent documentation. So I wanted to share my experience with you.

What will we cover in this article?

  • How to install the Falco agent on the host that you want to monitor for events (anomalies/violations)
  • How to tune Falco to reduce false positives and get the information you really need
  • How to use Prometheus to collect Falco events into a central location, with the help of the exporters and a scraper.
  • Finally, how to connect the scraper with Grafana for visualization and alerting

What do you need for this tutorial?

  • A machine or machines with Linux installed. A virtual machine should work.
  • You will need superuser permissions to be able to install/setup Docker, RPM, and systemd processes
  • We will use Docker containers, so basic knowledge of Docker is required
  • Working knowledge of Python/Bash, as we will write a few scripts to test and improve our configuration.

At the end you will be able to setup each one of the following components:

falco_monitoring

Don't be intimidated – I'll provide links to the documentation and a thorough explanation of each one of these tasks as we move along.

Table of contents

  1. What is Falco?
  2. How to Install Falco
  3. Basic Configuration
  4. How to Test the Default Configuration
  5. Defaults Are Not Always Good
  6. Falco Integrations
  7. Learning More

What is Falco?

The best way to describe this tool is to learn what it can do:

Falco can detect and alert on any behavior that involves making Linux system calls.

Falco alerts can be triggered by the use of specific system calls, their arguments, and by properties of the calling process. For example, Falco can easily detect incidents including but not limited to:

  • A shell is running inside a container or pod in Kubernetes.
  • A container is running in privileged mode, or is mounting a sensitive path, such as /proc, from the host.
  • A server process is spawning a child process of an unexpected type.
  • Unexpected read of a sensitive file, such as /etc/shadow.
  • A non-device file is written to /dev.
  • A standard system binary, such as ls, is making an outbound network connection.
  • A privileged pod is started in a Kubernetes cluster.

How to Install Falco

I will install Falco using an RPM (similar instructions exist for apt-get, and even Docker containers). In my case I felt the native installation was the best, and the RPM made it very easy to do:

[josevnz@macmini2 ~]$ sudo -i dnf install https://download.falco.org/packages/rpm/falco-0.31.1-x86_64.rpm
Last metadata expiration check: 2:53:53 ago on Sun 01 May 2022 04:13:09 PM EDT.
falco-0.31.1-x86_64.rpm                                                                                                                                                                                                       1.7 MB/s |  12 MB     00:07    
Dependencies resolved.
==============================================================================================================================================================================================================================================================
 Package                                                          Architecture                                      Version                                                                     Repository                                               Size
==============================================================================================================================================================================================================================================================
Installing:
 falco                                                            x86_64                                            0.31.1-1                                                                    @commandline                                             12 M
Installing dependencies:
 dkms                                                             noarch                                            2.8.1-4.20200214git5ca628c.fc30                                             updates                                                  78 k
 elfutils-libelf-devel                                            x86_64                                            0.179-2.fc30                                                                updates                                                  27 k
 kernel-devel                                                     x86_64                                            5.6.13-100.fc30                                                             updates                                                  14 M

Transaction Summary
==============================================================================================================================================================================================================================================================
Install  4 Packages

Total size: 26 M
Total download size: 14 M
Installed size: 92 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): elfutils-libelf-devel-0.179-2.fc30.x86_64.rpm                                                                                                                                                                          253 kB/s |  27 kB     00:00    
(2/3): dkms-2.8.1-4.20200214git5ca628c.fc30.noarch.rpm                                                                                                                                                                        342 kB/s |  78 kB     00:00    
(3/3): kernel-devel-5.6.13-100.fc30.x86_64.rpm                                                                                                                                                                                1.9 MB/s |  14 MB     00:07    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                         1.8 MB/s |  14 MB     00:07     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                                                                                      1/1 
  Installing       : kernel-devel-5.6.13-100.fc30.x86_64                                                                                                                                                                                                  1/4 
  Running scriptlet: kernel-devel-5.6.13-100.fc30.x86_64                                                                                                                                                                                                  1/4 
  Installing       : elfutils-libelf-devel-0.179-2.fc30.x86_64                                                                                                                                                                                            2/4 
  Installing       : dkms-2.8.1-4.20200214git5ca628c.fc30.noarch                                                                                                                                                                                          3/4 
  Running scriptlet: dkms-2.8.1-4.20200214git5ca628c.fc30.noarch                                                                                                                                                                                          3/4 
  Running scriptlet: falco-0.31.1-1.x86_64                                                                                                                                                                                                                4/4 
  Installing       : falco-0.31.1-1.x86_64                                                                                                                                                                                                                4/4 
  Running scriptlet: falco-0.31.1-1.x86_64                                  

Basic Configuration

Unless we want to do very basic output processing, we want to enable the JSON output:

# Whether to output events in json or text
json_output: true

It will become evident why pretty soon.

Next start the Falco agent:

[josevnz@macmini2 falco]$ sudo systemctl start falco.service 
[josevnz@macmini2 falco]$ sudo systemctl status falco.service 
● falco.service - Falco: Container Native Runtime Security
   Loaded: loaded (/usr/lib/systemd/system/falco.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-05-01 19:20:52 EDT; 1s ago
     Docs: https://falco.org/docs/
  Process: 26887 ExecStartPre=/sbin/modprobe falco (code=exited, status=0/SUCCESS)
 Main PID: 26888 (falco)
    Tasks: 1 (limit: 2310)
   Memory: 65.8M
   CGroup: /system.slice/falco.service
           └─26888 /usr/bin/falco --pidfile=/var/run/falco.pid

May 01 19:20:52 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 19:20:52 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 19:20:52 macmini2 falco[26888]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 19:20:52 macmini2 falco[26888]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 19:20:52 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 19:20:53 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 19:20:54 macmini2 falco[26888]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:

How to Test the Default Configuration

Depending on your configuration, you may or may not get any events right after starting Falco:

[josevnz@macmini2 falco]$ sudo journalctl --unit falco --follow
-- Logs begin at Tue 2021-05-25 00:15:22 EDT. --
May 01 19:20:52 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 19:20:52 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 19:20:52 macmini2 falco[26888]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 19:20:52 macmini2 falco[26888]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 19:20:52 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 19:20:53 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 19:20:54 macmini2 falco[26888]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
May 01 19:20:55 macmini2 falco[26888]: Starting internal webserver, listening on port 8765

Worry not. We will run a few commands that will cause Falco to record some warning and alerts. Time to see how this works!

How to Run a Privileged Container

Using privileged containers is considered a bad practice, so let's see if this event is detected by Falco:

[josevnz@macmini2 ~]$ docker run --rm --interactive --tty --privileged --volume /etc/shadow:/mnt/shadow fedora:latest ls -l /mnt/shadow
----------. 1 root root 1198 Nov 21 20:51 /mnt/shadow

And our Falco log?

May 01 19:29:32 macmini2 falco[26888]: {"output":"19:29:32.918828894: Informational Privileged container started (user=root user_loginuid=0 command=container:bfb9637a47a6 kind_lumiere (id=bfb9637a47a6) image=fedora:latest)","priority":"Informational","rule":"Launch Privileged Container","source":"syscall","tags":["cis","container","mitre_lateral_movement","mitre_privilege_escalation"],"time":"2022-05-01T23:29:32.918828894Z", "output_fields": {"container.id":"bfb9637a47a6","container.image.repository":"fedora","container.image.tag":"latest","container.name":"kind_lumiere","evt.time":1651447772918828894,"proc.cmdline":"container:bfb9637a47a6","user.loginuid":0,"user.name":"root"}}

It shows up as an informational event. Definitely one of those things to keep an eye on. Ask yourself if the application on the container needs elevated privileges.

You also probably noticed that each message has tags. Pay attention to the "mitre_*" ones, they do relate to the Mitre Attack knowledge base of attacks and mitigations. Yep, you will spend some time reading those.

How to Create a File on the /root Directory

This example shows how to abuse the root user combined with volumes in a container...

[josevnz@macmini2 ~]$ docker run --rm --interactive --tty --user root --volume /root:/mnt/ fedora:latest touch /mnt/test_file
[josevnz@macmini2 ~]$ 

Falco reaction:

May 01 19:32:02 macmini2 falco[26888]: {"output":"19:32:02.434286167: Informational Container with sensitive mount started (user=root user_loginuid=0 command=container:ef061174c7ef distracted_lalande (id=ef061174c7ef) image=fedora:latest mounts=/root:/mnt::true:rprivate)","priority":"Informational","rule":"Launch Sensitive Mount Container","source":"syscall","tags":["cis","container","mitre_lateral_movement"],"time":"2022-05-01T23:32:02.434286167Z", "output_fields": {"container.id":"ef061174c7ef","container.image.repository":"fedora","container.image.tag":"latest","container.mounts":"/root:/mnt::true:rprivate","container.name":"distracted_lalande","evt.time":1651447922434286167,"proc.cmdline":"container:ef061174c7ef","user.loginuid":0,"user.name":"root"}}

Sensitive mount detected!

Let's Raise the Stakes by Creating a File on /bin

Alright let's say we do this:

[josevnz@macmini2 ~]$ sudo -i
[root@macmini2 ~]# touch /bin/should_not_be_here

What does Falco think about it?

May 01 19:36:41 macmini2 falco[26888]: {"output":"19:36:41.237634398: Error File below a known binary directory opened for writing (user=root user_loginuid=1000 command=touch /bin/should_not_be_here file=/bin/should_not_be_here parent=bash pcmdline=bash gparent=sudo container_id=host image=<NA>)","priority":"Error","rule":"Write below binary dir","source":"syscall","tags":["filesystem","mitre_persistence"],"time":"2022-05-01T23:36:41.237634398Z", "output_fields": {"container.id":"host","container.image.repository":null,"evt.time":1651448201237634398,"fd.name":"/bin/should_not_be_here","proc.aname[2]":"sudo","proc.cmdline":"touch /bin/should_not_be_here","proc.pcmdline":"bash","proc.pname":"bash","user.loginuid":1000,"user.name":"root"}}

An error, binary directory opened for writing. Good catch.

Defaults Are Not Always Good

After Falco is running for a while it is a good idea to get a sense of what kind of events we want to ignore and which ones we want to investigate.

The first step is to get a list of all the events, using our JSON format on the payload:

sudo journalctl --unit falco --no-page --output=cat > /tmp/falco_json_lines.txt

The 'output=cat' tells journalctl to give us the message payload without timestamps (don't worry, the JSON message itself has timestamps).

Starting Falco: Container Native Runtime Security...
Started Falco: Container Native Runtime Security.
Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
Falco initialized with configuration file /etc/falco/falco.yaml
Loading rules from file /etc/falco/falco_rules.yaml:
Loading rules from file /etc/falco/falco_rules.local.yaml:
Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Starting internal webserver, listening on port 8765
{"output":"19:29:32.918828894: Informational Privileged container started (user=root user_loginuid=0 command=container:bfb9637a47a6 kind_lumiere (id=bfb9637a47a6) image=fedora:latest)","priority":"Informational","rule":"Launch Privileged Container","source":"syscall","tags":["cis","container","mitre_lateral_movement","mitre_privilege_escalation"],"time":"2022-05-01T23:29:32.918828894Z", "output_fields": {"container.id":"bfb9637a47a6","container.image.repository":"fedora","container.image.tag":"latest","container.name":"kind_lumiere","evt.time":1651447772918828894,"proc.cmdline":"container:bfb9637a47a6","user.loginuid":0,"user.name":"root"}}
{"output":"19:32:02.434286167: Informational Container with sensitive mount started (user=root user_loginuid=0 command=container:ef061174c7ef distracted_lalande (id=ef061174c7ef) image=fedora:latest mounts=/root:/mnt::true:rprivate)","priority":"Informational","rule":"Launch Sensitive Mount Container","source":"syscall","tags":["cis","container","mitre_lateral_movement"],"time":"2022-05-01T23:32:02.434286167Z", "output_fields": {"container.id":"ef061174c7ef","container.image.repository":"fedora","container.image.tag":"latest","container.mounts":"/root:/mnt::true:rprivate","container.name":"distracted_lalande","evt.time":1651447922434286167,"proc.cmdline":"container:ef061174c7ef","user.loginuid":0,"user.name":"root"}}

So far it looks interesting, but what about this?

{"output":"23:04:10.609949471: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=openat command=bash fd.name=/home/josevnz/.bash_history-01112.tmp name=/home/josevnz/.bash_history-01112.tmp path=<NA> oldpath=<NA> host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.609949471Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":"/home/josevnz/.bash_history-01112.tmp","evt.arg.oldpath":null,"evt.arg.path":null,"evt.time":1651633450609949471,"evt.type":"openat","fd.name":"/home/josevnz/.bash_history-01112.tmp","proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.635602857: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=openat command=bash fd.name=/home/josevnz/.bash_history-01627.tmp name=/home/josevnz/.bash_history-01627.tmp path=<NA> oldpath=<NA> host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.635602857Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":"/home/josevnz/.bash_history-01627.tmp","evt.arg.oldpath":null,"evt.arg.path":null,"evt.time":1651633450635602857,"evt.type":"openat","fd.name":"/home/josevnz/.bash_history-01627.tmp","proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.635851215: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=rename command=bash fd.name=<NA> name=<NA> path=<NA> oldpath=/home/josevnz/.bash_history-01627.tmp host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.635851215Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":null,"evt.arg.oldpath":"/home/josevnz/.bash_history-01627.tmp","evt.arg.path":null,"evt.time":1651633450635851215,"evt.type":"rename","fd.name":null,"proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.661829867: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=rename command=bash fd.name=<NA> name=<NA> path=<NA> oldpath=/home/josevnz/.bash_history-01112.tmp host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.661829867Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":null,"evt.arg.oldpath":"/home/josevnz/.bash_history-01112.tmp","evt.arg.path":null,"evt.time":1651633450661829867,"evt.type":"rename","fd.name":null,"proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}

This is a normal/ legitimate operation. Let's find a way to harden this rule or remove it completely.

First, open the /etc/falco/falco_rules.yaml file and look for the rule 'Delete or rename shell history' (JSON output we saw earlier):

- list: docker_binaries
  items: [docker, dockerd, exe, docker-compose, docker-entrypoi, docker-runc-cur, docker-current, dockerd-current]

 macro: var_lib_docker_filepath
  condition: (evt.arg.name startswith /var/lib/docker or fd.name startswith /var/lib/docker)

- rule: Delete or rename shell history
  desc: Detect shell history deletion
  condition: >
    (modify_shell_history or truncate_shell_history) and
       not var_lib_docker_filepath and
       not proc.name in (docker_binaries)
  output: >
    Shell history had been deleted or renamed (user=%user.name user_loginuid=%user.loginuid type=%evt.type command=%proc.cmdline fd.name=%fd.name name=%evt.arg.name path=%evt.arg.path oldpath=%evt.arg.oldpath %container.info)
  priority:
    WARNING
  tags: [process, mitre_defense_evasion]

Falco rules are explained in detail on the official documentation. Just by looking at this piece you will notice a few things.

About the conditions:

  1. Support complex logic,
  2. macros like var_lib_docker_filepath
  3. lists like (docker_binaries)
  4. and special variables with fields like proc.name.

It is recommended that do you not change this file. Instead you should override what you need on the /etc/falco/falco_rules.local.yaml:

# Add new rules, like this one
# - rule: The program "sudo" is run in a container
#   desc: An event will trigger every time you run sudo in a container
#   condition: evt.type = execve and evt.dir=< and container.id != host and proc.name = sudo
#   output: "Sudo run in container (user=%user.name %container.info parent=%proc.pname cmdline=%proc.cmdline)"
#   priority: ERROR
#   tags: [users, container]

# Or override/append to any rule, macro, or list from the Default Rules

For the sake of example, say that we do care when the history of the super-user (root) is overridden, but everybody else if fine. The best part is that you don't have to override the whole rule.

So the original rule will get a condition appended:

- rule: Delete or rename shell history
  append: true
  condition: and user.name=root

It is always a good idea to validate that your rules are properly written. For that you need can tell Falco to check the original rules and your overrides together:

[root@macmini2 ~]# falco --validate /etc/falco/falco_rules.yaml --validate /etc/falco/falco_rules.local.yaml 
Fri May  6 20:48:00 2022: Validating rules file(s):
Fri May  6 20:48:00 2022:    /etc/falco/falco_rules.yaml
Fri May  6 20:48:00 2022:    /etc/falco/falco_rules.local.yaml
/etc/falco/falco_rules.yaml: Ok
/etc/falco/falco_rules.local.yaml: Ok
Fri May  6 20:48:01 2022: Ok

# If the rules are OK, restart Falco
[root@macmini2 ~]# systemctl restart falco.service

How to Make a Simple Event Explorer in Python

You'll probably agree that getting a sense of what rules are noise and which ones are useful is tedious.

We need to normalize this data, and we will use a Python script that will:

  • Remove non-JSON data
  • Aggregate event types without the timestamps
  • Generate a few aggregation statistics, so we can focus on the most frequent events in our system

A small Python script can do the trick. I'm leaving out the UI rendering part (please check the code to see the full picture), and instead will show you the file parsing bits:

#!/usr/bin/env python3
"""
Aggregate Falco events to make it easier to override rules
Jose Vicente Nunez (kodegeek.com@protonmail.com)
"""
import json
import re
from argparse import ArgumentParser
from pathlib import Path
from rich.console import Console
from falcotutor.ui import EventDisplayApp, create_event_table, add_rows_to_create_event_table


def filter_events(journalctl_out: Path) -> dict[any, any]:
    """
    :param journalctl_out:
    :return:
    """
    with open(journalctl_out, 'r') as journalctl_file:
        for row in journalctl_file:
            if re.search("^{", row):
                data = json.loads(row)
                if 'rule' in data and 'output_fields' in data:
                    yield data


def aggregate_events(local_event: dict[any, any], aggregated_events: dict[any, any]):
    rule = local_event['rule']
    if rule not in aggregated_events:
        aggregated_events[rule] = {
            'count': 0,
            'priority': local_event['priority'],
            'last_timestamp': "",
            'last_fields': ""
        }
    aggregated_events[rule]['count'] += 1
    aggregated_events[rule]['last_timestamp'] = local_event['time']
    del local_event['output_fields']['evt.time']
    aggregated_events[rule]['last_fields'] = json.dumps(local_event['output_fields'], indent=True)


if __name__ == "__main__":
    CONSOLE = Console()
    AGGREGATED = {}
    PARSER = ArgumentParser(description=__doc__)
    PARSER.add_argument(
        "falco_event",
        action="store"
    )
    ARGS = PARSER.parse_args()
    try:
        event_table = create_event_table()
        for event in filter_events(ARGS.falco_event):
            aggregate_events(local_event=event, aggregated_events=AGGREGATED)
        add_rows_to_create_event_table(AGGREGATED, event_table)
        EventDisplayApp.run(
            event_file=ARGS.falco_event,
            title="Falco aggregated events report",
            event_table=event_table
        )
    except KeyboardInterrupt:
        CONSOLE.print("[bold]Program interrupted...[/bold]")

Once the file is loaded as a dictionary, we only need to iterate to it to aggregate the events, then show the results as a neat table sorted by count:

asciicast

How to Show the Falco Rules

If you are like me, you are always looking at the /etc/falco/falco_rules.yaml file to understand what is being monitored. A brief view of those rules (without looking at the verbose YAML file with comments) is a nice addition:

#!/usr/bin/env python3
"""
Show brief content of default Falco rule YAML files
Jose Vicente Nunez (kodegeek.com@protonmail.com)
"""
from argparse import ArgumentParser
from pathlib import Path
from rich.console import Console
import yaml
from falcotutor.ui import create_rules_table, add_rows_to_create_rules_table, RulesDisplayApp


def load_rulez(falco_rulez: Path) -> dict[any, any]:
    rulez = {}
    with open(falco_rulez, 'rt') as falco_file:
        for rule_data in yaml.full_load(falco_file):
            if 'rule' in rule_data:
                rule_name = rule_data['rule']
                del rule_data['rule']
                rulez[rule_name] = rule_data
    return rulez


if __name__ == "__main__":
    CONSOLE = Console()
    AGGREGATED = {}
    PARSER = ArgumentParser(description=__doc__)
    PARSER.add_argument(
        "falco_rules",
        action="store"
    )
    ARGS = PARSER.parse_args()
    try:
        RULES = load_rulez(ARGS.falco_rules)
        RULE_TBL = create_rules_table()
        add_rows_to_create_rules_table(lrules=RULES, rules_tbl=RULE_TBL)
        RulesDisplayApp.run(
            rules_file=ARGS.falco_rules,
            title="Falco brief rule display",
            rules_table=RULE_TBL
        )
    except KeyboardInterrupt:
        CONSOLE.print("[bold]Program interrupted...[/bold]")

You could improve this script by adding rule filtering by certain criteria, for example (rule name, priority, enabled/ disabled). This version doesn't do any filtering:

asciicast

Falco Integrations

You probably noticed two things from our earlier experimentation:

  1. The payload of the events do not have the host. If you want to locate an offending server, you need to improve how a multi-host event is reported (parsing a journalctl file from many hosts is not practical).
  2. We want to get alerts in a centralized location. It would be nice to have a way to "push" those events instead of us going to fish.

It is time to consolidate those alerts in a single place.

How to Use Falco Exporter

The Falco exporter will allow us to share the Falco alerts with Prometheus scraper. We need first to enable gRPC in the /etc/falco/falco.yaml

# gRPC server using an unix socket
grpc:
  enabled: true
  bind_address: "unix:///var/run/falco.sock"
  # when threadiness is 0, Falco automatically guesses it depending on the number of online cores
  threadiness: 0

# gRPC output service.
# By default it is off.
# By enabling this all the output events will be kept in memory until you read them with a gRPC client.
# Make sure to have a consumer for them or leave this disabled.
grpc_output:
  enabled: true

Restart Falco:

[root@macmini2 ~]# systemctl restart falco.service 
[root@macmini2 ~]# systemctl status falco.service 
● falco.service - Falco: Container Native Runtime Security
   Loaded: loaded (/usr/lib/systemd/system/falco.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2022-05-01 20:35:01 EDT; 26s ago
     Docs: https://falco.org/docs/
  Process: 28285 ExecStartPre=/sbin/modprobe falco (code=exited, status=0/SUCCESS)
 Main PID: 28288 (falco)
    Tasks: 11 (limit: 2310)
   Memory: 80.9M
   CGroup: /system.slice/falco.service
           └─28288 /usr/bin/falco --pidfile=/var/run/falco.pid

May 01 20:35:01 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 20:35:01 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 20:35:01 macmini2 falco[28288]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 20:35:01 macmini2 falco[28288]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 20:35:01 macmini2 falco[28288]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 20:35:02 macmini2 falco[28288]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 20:35:03 macmini2 falco[28288]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
May 01 20:35:04 macmini2 falco[28288]: Starting internal webserver, listening on port 8765
May 01 20:35:04 macmini2 falco[28288]: gRPC server threadiness equals to 2
May 01 20:35:04 macmini2 falco[28288]: Starting gRPC server at unix:///var/run/falco.sock

Quickly make sure everything is OK (reminder, the Falco agent is running on macmini2):

josevnz@raspberrypi:~$ curl --fail http://macmini2:8765/healthz
{"status": "ok"}josevnz@raspberrypi:~$

Then we run the falco-exporter. To make it easier, we will use a Docker container with a few overrides in the command line.

[root@macmini2 ~]# docker run --restart always --name falco-exporter --detach --volume /var/run/falco.sock:/var/run/falco.sock --network=host falcosecurity/falco-exporter --listen-address 192.168.1.16:9376
7d157af0251ea4bc73b8c355a74eaf4dd24a5348cbe3f5f2ea9d7147c6c366c8
[root@macmini2 ~]# docker logs falco-exporter
2022/05/02 00:56:30 connecting to gRPC server at unix:///var/run/falco.sock (timeout 2m0s)
2022/05/02 00:56:30 listening on http://192.168.1.16:9376/metrics
2022/05/02 00:56:30 connected to gRPC server, subscribing events stream
2022/05/02 00:56:30 ready

# Check with CURL if the URL is reachable
[root@macmini2 ~]# curl http://192.168.1.16:9376/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 18
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.14.15"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.011112e+06

For completeness, let me show you also how to capture the host performance metrics using the node exporter (we will use it later to keep an eye on how many resources are used by Falco and to make sure our installation is not hurting the server):

docker run --detach --net="host" --pid="host" --volume "/:/host:ro,rslave" quay.io/prometheus/node-exporter:latest --path.rootfs=/host

The node-exporter and the falco-exporter will run on every host that needs their data scraped. Now you need to wait to collect all these metrics into a single location. For that will use the Prometheus agent:

---
# /etc/prometheus.yaml on raspberrypi
global:
    scrape_interval: 30s
    evaluation_interval: 30s
    scrape_timeout: 10s
    external_labels:
        monitor: 'nunez-family-monitor'

scrape_configs:
  - job_name: 'falco-exporter'
    static_configs:
      - targets: ['macmini2.home:9376']
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['macmini2.home:9100', 'raspberrypi.home:9100', 'dmaf5:9100']
  - job_name: 'docker-exporter'
    static_configs:
      - targets: ['macmini2.home:9323', 'raspberrypi.home:9323', 'dmaf5:9323']

    tls_config:
      insecure_skip_verify: true

Then make sure the Prometheus scraper can talk with each one of the nodes. We visit the web UI:

prometheus-raspberrypi

Good, Prometheus is able to scrape Falco. We can even run a simple query to see a few events:

prometheus-query-falco

Next we need to setup the UI view for the events, and for that we will use Grafana.

There are many ways to install Grafana. In my case I will use a Grafana Docker container (I will run Grafana on the same host where Prometheus is running: raspberripi.home):

docker pull grafana/grafana:main-ubuntu
mkdir -p /data/grafana
chown syslog /data/grafana
docker run --user 104 --name grafana --detach --tty --volume /data/grafana:/var/lib/grafana -p 3000:3000 grafana/grafana:main-ubuntu

After Grafana comes up, you will need to change your password and will also need to connect with Prometheus:

grafana-prometheus-falco

Once Grafana is up, we can import the Falco dashboard as explained here.

falco-grafana-integration

Once the dashboard is imported we can generate a few events to trigger Falco on the host where is installed:

[root@macmini2 ~]# for i in $(seq 1 60); do docker run --rm --interactive --tty --privileged fedora:latest /bin/bash -c ls; touch /root/test; rm -f /root/test; sleep 1; done

After a little you should see something like this on your Grafana Dashboard:

grafana-falco-dashboard

The events are flowing, and you can see from which host they came.

How to Create Alerts for Your Falco Events

Ideally if you have the Falco events in Grafana, you can make these actionable items and generate alerts from then.

I don't want to get bombarded by non-critical alerts, so the first thing to know is to what level of events to filter:

 

grafana-priority-events

Anything with priority below 3 will be treated as an alert.

Grafana has good documentation on how to setup an alert, so I will show here the end result only:

grafana-falco-alert-definition

The next step is to send the alerts somewhere.

Alerts Need to Go Somewhere – How to Define a Contact Point using Discord

For this example we will use Discord as the end for the alerts. Discord has a very detailed guide on how to setup a WebHook, so I will only show you here the end results of my discord Webhook:

grafana-discord-webhook

We copy that URL and then will configure a new Grafana contact point that uses our Discord webhook (we are setting this a default contact point for all the alerts):

grafana-contact-point-discord

From there we can send a test message to Discord, just to confirm that this pipeline works:

grafana-discord-events-test

We're getting closer. By now if we go back to our alert definition we should see it is on the 'firing' state:

grafana-falco-alert-firing

And if everything goes well we see also our first Falco alert in Discord:

grafana-falco-discord-alert

We can see here all the fields we get on the journalctl output. The difference is that all these messages will come from all the servers where you define the Falco-Prometheus-Grafana bridge.

Honorable Mention: How to Aggregate Alerts using Falcon Sidekick/ Falcon Sidekick-UI

Falco Sidekick is another way to gather and send events to other destinations, like the Falco Sidekick-UI. But it won't tell you the originating host (at least until Falco 0.31.1).

This is most likely not an issue for an alert coming from a K8s cluster or a containerized application where the image name will give you plenty of information. But if your event happens on a bare-metal environment, and you have more than 2 machines, it will become a headache.

For that reason I won't cover Sidekick here – you may want to stick with the Grafana integration for the time being.

  Source: https://www.freecodecamp.org/news/secure-server-infrastructure-clouds-using-falco-prometheus-grafana-and-docker/

#falco #grafana #docker #cloud 

Grafana dashboard tutorial: custom data visualization with Grafana + Prometheus

You may already know how important it is to measure the performance of your app, or even how to integrate with tools such as Prometheus. But if you can’t visualize your data in an easy-to-read and organized manner, you won’t get much out of it. There’s a cool, open-source tool for that. In the last part of the Metrics in optimization process series, we’re teaching you how to create a dashboard in Grafana, so that you can produce a new dashboard that even the business department will follow.

What you will learn here

In the previous three articles, we talked about why you should measure your app’s metrics, how you can integrate it with Prometheus and how to go about collecting statistics.

Since we have now a lot of data to show we should think about the presentation layer. The true purpose of it is to present data in a way that gives us what we need, rather than just as a disorganized pile of trash. It will not just make pretties, but also far more useful. Another important thing is that the data, which we collected with Prometheus, doesn’t have an easy-to-read format. Today it’s time to show ugly metrics data in a beautiful way.

As you know from previous articles, we collect data from Kubernetes and our application. We are going to show an example of how to configure a dashboard, what is important, and why we did it this way. We are not going to show and describe all options in Grafana, just the ones that we needed to configure our charts (hey, it’s a crash course, after all!).

#developer stories #data #devops #grafana #kubernetes #prometheus

Myah  Conn

Myah Conn

1591101480

Getting Started with Prometheus and Grafana

This post is broken up into 3 separate sections : “Introduction”, “Setting up Prometheus” and “Setting up Grafana For Prometheus”. The First section will concern the explanation of System Monitoring, Prometheus, and its different components. The second section will deal with the installation and the setup up part of Prometheus on Ubuntu 18.04. The last section will deal with the configuration and the integration of Grafana to Prometheus

#grafana #devops #ubuntu #monitoring #prometheus