1652299200
Falco、Prometheus、Grafana、Dockerを使用してサーバーインフラストラクチャクラウドを保護する
私は最近、コンテナとアプリケーションを機能させ続ける方法を探していました。具体的には、構成の異常を検出することに興味がありました。少し調べた後、私はFalcoに出くわしました。
私が見つけたのは、多くの機能と優れたドキュメントを備えた非常に完全なオープンソースプラットフォームでした。だから私はあなたと私の経験を共有したいと思いました。
この記事では何を取り上げますか？
- イベント（異常/違反）を監視するホストにFalcoエージェントをインストールする方法
- Falcoを調整して誤検知を減らし、本当に必要な情報を取得する方法
- Prometheusを使用して、輸出業者とスクレーパーの助けを借りて、Falcoイベントを中央の場所に収集する方法。
- 最後に、視覚化とアラートのためにスクレーパーをGrafanaに接続する方法
このチュートリアルには何が必要ですか？
- Linuxがインストールされている1つまたは複数のマシン。仮想マシンが機能するはずです。
- Docker、RPM、およびsystemdプロセスをインストール/セットアップできるようにするには、スーパーユーザー権限が必要です。
- Dockerコンテナーを使用するため、Dockerの基本的な知識が必要です。
- 構成をテストおよび改善するためのスクリプトをいくつか作成するため、Python/Bashの実用的な知識。
最後に、次の各コンポーネントをセットアップできるようになります。
恐れることはありません。ドキュメントへのリンクと、これらの各タスクの詳細な説明を提供します。
目次
ファルコとは？
このツールを説明する最良の方法は、ツールで何ができるかを学ぶことです。
Falcoは、Linuxシステムコールの実行に関連するすべての動作を検出して警告することができます。
Falcoアラートは、特定のシステムコールとその引数の使用、および呼び出しプロセスのプロパティによってトリガーできます。たとえば、Falcoは、以下を含むがこれらに限定されないインシデントを簡単に検出できます。
- Kubernetesのコンテナまたはポッド内でシェルが実行されています。
- コンテナが特権モードで実行されているか、ホストから/procなどの機密パスをマウントしています。
- サーバープロセスは、予期しないタイプの子プロセスを生成しています。
- / etc/shadowなどの機密ファイルの予期しない読み取り。
- デバイス以外のファイルが/devに書き込まれます。
- lsなどの標準のシステムバイナリは、アウトバウンドネットワーク接続を確立しています。
- 特権ポッドがKubernetesクラスターで開始されます。
Falcoのインストール方法
RPMを使用してFalcoをインストールします（apt-get、さらにはDockerコンテナーにも同様の手順があります）。私の場合、ネイティブインストールが最適であり、RPMによって非常に簡単に実行できると感じました。
[josevnz@macmini2 ~]$ sudo -i dnf install https://download.falco.org/packages/rpm/falco-0.31.1-x86_64.rpm
Last metadata expiration check: 2:53:53 ago on Sun 01 May 2022 04:13:09 PM EDT.
falco-0.31.1-x86_64.rpm 1.7 MB/s | 12 MB 00:07
Dependencies resolved.
==============================================================================================================================================================================================================================================================
Package Architecture Version Repository Size
==============================================================================================================================================================================================================================================================
Installing:
falco x86_64 0.31.1-1 @commandline 12 M
Installing dependencies:
dkms noarch 2.8.1-4.20200214git5ca628c.fc30 updates 78 k
elfutils-libelf-devel x86_64 0.179-2.fc30 updates 27 k
kernel-devel x86_64 5.6.13-100.fc30 updates 14 M
Transaction Summary
==============================================================================================================================================================================================================================================================
Install 4 Packages
Total size: 26 M
Total download size: 14 M
Installed size: 92 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): elfutils-libelf-devel-0.179-2.fc30.x86_64.rpm 253 kB/s | 27 kB 00:00
(2/3): dkms-2.8.1-4.20200214git5ca628c.fc30.noarch.rpm 342 kB/s | 78 kB 00:00
(3/3): kernel-devel-5.6.13-100.fc30.x86_64.rpm 1.9 MB/s | 14 MB 00:07
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 1.8 MB/s | 14 MB 00:07
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : kernel-devel-5.6.13-100.fc30.x86_64 1/4
Running scriptlet: kernel-devel-5.6.13-100.fc30.x86_64 1/4
Installing : elfutils-libelf-devel-0.179-2.fc30.x86_64 2/4
Installing : dkms-2.8.1-4.20200214git5ca628c.fc30.noarch 3/4
Running scriptlet: dkms-2.8.1-4.20200214git5ca628c.fc30.noarch 3/4
Running scriptlet: falco-0.31.1-1.x86_64 4/4
Installing : falco-0.31.1-1.x86_64 4/4
Running scriptlet: falco-0.31.1-1.x86_64
基本構成
非常に基本的な出力処理を行う場合を除いて、JSON出力を有効にします。
# Whether to output events in json or text
json_output: true
理由はすぐに明らかになります。
次に、Falcoエージェントを開始します。
[josevnz@macmini2 falco]$ sudo systemctl start falco.service
[josevnz@macmini2 falco]$ sudo systemctl status falco.service
● falco.service - Falco: Container Native Runtime Security
Loaded: loaded (/usr/lib/systemd/system/falco.service; disabled; vendor preset: disabled)
Active: active (running) since Sun 2022-05-01 19:20:52 EDT; 1s ago
Docs: https://falco.org/docs/
Process: 26887 ExecStartPre=/sbin/modprobe falco (code=exited, status=0/SUCCESS)
Main PID: 26888 (falco)
Tasks: 1 (limit: 2310)
Memory: 65.8M
CGroup: /system.slice/falco.service
└─26888 /usr/bin/falco --pidfile=/var/run/falco.pid
May 01 19:20:52 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 19:20:52 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 19:20:52 macmini2 falco[26888]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 19:20:52 macmini2 falco[26888]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 19:20:52 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 19:20:53 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 19:20:54 macmini2 falco[26888]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
デフォルト設定をテストする方法
構成によっては、Falcoを起動した直後にイベントが発生する場合と発生しない場合があります。
[josevnz@macmini2 falco]$ sudo journalctl --unit falco --follow
-- Logs begin at Tue 2021-05-25 00:15:22 EDT. --
May 01 19:20:52 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 19:20:52 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 19:20:52 macmini2 falco[26888]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 19:20:52 macmini2 falco[26888]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 19:20:52 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 19:20:53 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 19:20:54 macmini2 falco[26888]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
May 01 19:20:55 macmini2 falco[26888]: Starting internal webserver, listening on port 8765
心配しないでください。Falcoに警告とアラートを記録させるいくつかのコマンドを実行します。これがどのように機能するかを見てみましょう！
特権コンテナを実行する方法
特権コンテナの使用は悪い習慣と見なされているので、このイベントがFalcoによって検出されるかどうかを見てみましょう。
[josevnz@macmini2 ~]$ docker run --rm --interactive --tty --privileged --volume /etc/shadow:/mnt/shadow fedora:latest ls -l /mnt/shadow
----------. 1 root root 1198 Nov 21 20:51 /mnt/shadow
そして私たちのファルコログ？
May 01 19:29:32 macmini2 falco[26888]: {"output":"19:29:32.918828894: Informational Privileged container started (user=root user_loginuid=0 command=container:bfb9637a47a6 kind_lumiere (id=bfb9637a47a6) image=fedora:latest)","priority":"Informational","rule":"Launch Privileged Container","source":"syscall","tags":["cis","container","mitre_lateral_movement","mitre_privilege_escalation"],"time":"2022-05-01T23:29:32.918828894Z", "output_fields": {"container.id":"bfb9637a47a6","container.image.repository":"fedora","container.image.tag":"latest","container.name":"kind_lumiere","evt.time":1651447772918828894,"proc.cmdline":"container:bfb9637a47a6","user.loginuid":0,"user.name":"root"}}
情報イベントとして表示されます。間違いなく注目すべきものの1つです。コンテナ上のアプリケーションに昇格された特権が必要かどうかを自問してください。
また、各メッセージにタグが付いていることにお気づきかもしれません。「mitre_*」のものに注意してください。これらは、攻撃と軽減に関するMitreAttackナレッジベースに関連しています。うん、あなたはそれらを読むのに少し時間を費やすでしょう。
/rootディレクトリにファイルを作成する方法
この例は、コンテナ内のボリュームと組み合わせたrootユーザーを悪用する方法を示しています...
[josevnz@macmini2 ~]$ docker run --rm --interactive --tty --user root --volume /root:/mnt/ fedora:latest touch /mnt/test_file
[josevnz@macmini2 ~]$
ファルコ反応：
May 01 19:32:02 macmini2 falco[26888]: {"output":"19:32:02.434286167: Informational Container with sensitive mount started (user=root user_loginuid=0 command=container:ef061174c7ef distracted_lalande (id=ef061174c7ef) image=fedora:latest mounts=/root:/mnt::true:rprivate)","priority":"Informational","rule":"Launch Sensitive Mount Container","source":"syscall","tags":["cis","container","mitre_lateral_movement"],"time":"2022-05-01T23:32:02.434286167Z", "output_fields": {"container.id":"ef061174c7ef","container.image.repository":"fedora","container.image.tag":"latest","container.mounts":"/root:/mnt::true:rprivate","container.name":"distracted_lalande","evt.time":1651447922434286167,"proc.cmdline":"container:ef061174c7ef","user.loginuid":0,"user.name":"root"}}
敏感なマウントが検出されました！
/binにファイルを作成して賭け金を上げましょう
さて、これを行うとしましょう：
[josevnz@macmini2 ~]$ sudo -i
[root@macmini2 ~]# touch /bin/should_not_be_here
ファルコはそれについてどう思いますか？
May 01 19:36:41 macmini2 falco[26888]: {"output":"19:36:41.237634398: Error File below a known binary directory opened for writing (user=root user_loginuid=1000 command=touch /bin/should_not_be_here file=/bin/should_not_be_here parent=bash pcmdline=bash gparent=sudo container_id=host image=<NA>)","priority":"Error","rule":"Write below binary dir","source":"syscall","tags":["filesystem","mitre_persistence"],"time":"2022-05-01T23:36:41.237634398Z", "output_fields": {"container.id":"host","container.image.repository":null,"evt.time":1651448201237634398,"fd.name":"/bin/should_not_be_here","proc.aname[2]":"sudo","proc.cmdline":"touch /bin/should_not_be_here","proc.pcmdline":"bash","proc.pname":"bash","user.loginuid":1000,"user.name":"root"}}
エラー、書き込み用に開かれたバイナリディレクトリ。良いキャッチ。
デフォルトは必ずしも良いとは限りません
Falcoがしばらく実行された後、どのような種類のイベントを無視し、どのイベントを調査したいのかを把握することをお勧めします。
最初のステップは、ペイロードでJSON形式を使用して、すべてのイベントのリストを取得することです。
sudo journalctl --unit falco --no-page --output=cat > /tmp/falco_json_lines.txt
'output = cat'は、タイムスタンプなしでメッセージペイロードを提供するようにjournalctlに指示します（心配しないでください。JSONメッセージ自体にタイムスタンプがあります）。
Starting Falco: Container Native Runtime Security...
Started Falco: Container Native Runtime Security.
Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
Falco initialized with configuration file /etc/falco/falco.yaml
Loading rules from file /etc/falco/falco_rules.yaml:
Loading rules from file /etc/falco/falco_rules.local.yaml:
Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Starting internal webserver, listening on port 8765
{"output":"19:29:32.918828894: Informational Privileged container started (user=root user_loginuid=0 command=container:bfb9637a47a6 kind_lumiere (id=bfb9637a47a6) image=fedora:latest)","priority":"Informational","rule":"Launch Privileged Container","source":"syscall","tags":["cis","container","mitre_lateral_movement","mitre_privilege_escalation"],"time":"2022-05-01T23:29:32.918828894Z", "output_fields": {"container.id":"bfb9637a47a6","container.image.repository":"fedora","container.image.tag":"latest","container.name":"kind_lumiere","evt.time":1651447772918828894,"proc.cmdline":"container:bfb9637a47a6","user.loginuid":0,"user.name":"root"}}
{"output":"19:32:02.434286167: Informational Container with sensitive mount started (user=root user_loginuid=0 command=container:ef061174c7ef distracted_lalande (id=ef061174c7ef) image=fedora:latest mounts=/root:/mnt::true:rprivate)","priority":"Informational","rule":"Launch Sensitive Mount Container","source":"syscall","tags":["cis","container","mitre_lateral_movement"],"time":"2022-05-01T23:32:02.434286167Z", "output_fields": {"container.id":"ef061174c7ef","container.image.repository":"fedora","container.image.tag":"latest","container.mounts":"/root:/mnt::true:rprivate","container.name":"distracted_lalande","evt.time":1651447922434286167,"proc.cmdline":"container:ef061174c7ef","user.loginuid":0,"user.name":"root"}}
これまでのところ面白そうですが、これはどうですか？
{"output":"23:04:10.609949471: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=openat command=bash fd.name=/home/josevnz/.bash_history-01112.tmp name=/home/josevnz/.bash_history-01112.tmp path=<NA> oldpath=<NA> host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.609949471Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":"/home/josevnz/.bash_history-01112.tmp","evt.arg.oldpath":null,"evt.arg.path":null,"evt.time":1651633450609949471,"evt.type":"openat","fd.name":"/home/josevnz/.bash_history-01112.tmp","proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.635602857: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=openat command=bash fd.name=/home/josevnz/.bash_history-01627.tmp name=/home/josevnz/.bash_history-01627.tmp path=<NA> oldpath=<NA> host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.635602857Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":"/home/josevnz/.bash_history-01627.tmp","evt.arg.oldpath":null,"evt.arg.path":null,"evt.time":1651633450635602857,"evt.type":"openat","fd.name":"/home/josevnz/.bash_history-01627.tmp","proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.635851215: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=rename command=bash fd.name=<NA> name=<NA> path=<NA> oldpath=/home/josevnz/.bash_history-01627.tmp host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.635851215Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":null,"evt.arg.oldpath":"/home/josevnz/.bash_history-01627.tmp","evt.arg.path":null,"evt.time":1651633450635851215,"evt.type":"rename","fd.name":null,"proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.661829867: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=rename command=bash fd.name=<NA> name=<NA> path=<NA> oldpath=/home/josevnz/.bash_history-01112.tmp host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.661829867Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":null,"evt.arg.oldpath":"/home/josevnz/.bash_history-01112.tmp","evt.arg.path":null,"evt.time":1651633450661829867,"evt.type":"rename","fd.name":null,"proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
これは通常の/正当な操作です。このルールを強化するか、完全に削除する方法を見つけましょう。
まず、/etc/falco/falco_rules.yamlファイルを開き、ルール「シェル履歴の削除または名前変更」（前に見たJSON出力）を探します。
- list: docker_binaries
items: [docker, dockerd, exe, docker-compose, docker-entrypoi, docker-runc-cur, docker-current, dockerd-current]
macro: var_lib_docker_filepath
condition: (evt.arg.name startswith /var/lib/docker or fd.name startswith /var/lib/docker)
- rule: Delete or rename shell history
desc: Detect shell history deletion
condition: >
(modify_shell_history or truncate_shell_history) and
not var_lib_docker_filepath and
not proc.name in (docker_binaries)
output: >
Shell history had been deleted or renamed (user=%user.name user_loginuid=%user.loginuid type=%evt.type command=%proc.cmdline fd.name=%fd.name name=%evt.arg.name path=%evt.arg.path oldpath=%evt.arg.oldpath %container.info)
priority:
WARNING
tags: [process, mitre_defense_evasion]
Falcoのルールは、公式ドキュメントで詳細に説明されています。この作品を見るだけで、いくつかのことに気付くでしょう。
条件について：
- 複雑なロジックをサポートし、
- のようなマクロ
var_lib_docker_filepath - のようなリスト
(docker_binaries) - のようなフィールドを持つ特別な変数
proc.name。
このファイルは変更しないことをお勧めします。/etc/falco/falco_rules.local.yaml代わりに、 ：で必要なものをオーバーライドする必要があります。
# Add new rules, like this one
# - rule: The program "sudo" is run in a container
# desc: An event will trigger every time you run sudo in a container
# condition: evt.type = execve and evt.dir=< and container.id != host and proc.name = sudo
# output: "Sudo run in container (user=%user.name %container.info parent=%proc.pname cmdline=%proc.cmdline)"
# priority: ERROR
# tags: [users, container]
# Or override/append to any rule, macro, or list from the Default Rules
例として、スーパーユーザー（root）の履歴が上書きされる場合は注意しますが、それ以外の場合は問題ないとします。最良の部分は、ルール全体をオーバーライドする必要がないことです。
したがって、元のルールには条件が追加されます。
- rule: Delete or rename shell history
append: true
condition: and user.name=root
ルールが適切に記述されていることを検証することは常に良い考えです。そのためには、元のルールとオーバーライドを一緒にチェックするようにFalcoに指示する必要があります。
[root@macmini2 ~]# falco --validate /etc/falco/falco_rules.yaml --validate /etc/falco/falco_rules.local.yaml
Fri May 6 20:48:00 2022: Validating rules file(s):
Fri May 6 20:48:00 2022: /etc/falco/falco_rules.yaml
Fri May 6 20:48:00 2022: /etc/falco/falco_rules.local.yaml
/etc/falco/falco_rules.yaml: Ok
/etc/falco/falco_rules.local.yaml: Ok
Fri May 6 20:48:01 2022: Ok
# If the rules are OK, restart Falco
[root@macmini2 ~]# systemctl restart falco.service
Pythonで簡単なイベントエクスプローラーを作成する方法
どのルールがノイズであり、どのルールが有用であるかを理解するのは面倒であることにおそらく同意するでしょう。
このデータを正規化する必要があり、次のようなPythonスクリプトを使用します。
- 非JSONデータを削除する
- タイムスタンプなしでイベントタイプを集約します
- いくつかの集計統計を生成して、システムで最も頻繁に発生するイベントに集中できるようにします
小さなPythonスクリプトでうまくいくことができます。UIレンダリング部分を省略し（コードを確認して全体像を確認してください）、代わりにファイル解析ビットを表示します。
#!/usr/bin/env python3
"""
Aggregate Falco events to make it easier to override rules
Jose Vicente Nunez (kodegeek.com@protonmail.com)
"""
import json
import re
from argparse import ArgumentParser
from pathlib import Path
from rich.console import Console
from falcotutor.ui import EventDisplayApp, create_event_table, add_rows_to_create_event_table
def filter_events(journalctl_out: Path) -> dict[any, any]:
"""
:param journalctl_out:
:return:
"""
with open(journalctl_out, 'r') as journalctl_file:
for row in journalctl_file:
if re.search("^{", row):
data = json.loads(row)
if 'rule' in data and 'output_fields' in data:
yield data
def aggregate_events(local_event: dict[any, any], aggregated_events: dict[any, any]):
rule = local_event['rule']
if rule not in aggregated_events:
aggregated_events[rule] = {
'count': 0,
'priority': local_event['priority'],
'last_timestamp': "",
'last_fields': ""
}
aggregated_events[rule]['count'] += 1
aggregated_events[rule]['last_timestamp'] = local_event['time']
del local_event['output_fields']['evt.time']
aggregated_events[rule]['last_fields'] = json.dumps(local_event['output_fields'], indent=True)
if __name__ == "__main__":
CONSOLE = Console()
AGGREGATED = {}
PARSER = ArgumentParser(description=__doc__)
PARSER.add_argument(
"falco_event",
action="store"
)
ARGS = PARSER.parse_args()
try:
event_table = create_event_table()
for event in filter_events(ARGS.falco_event):
aggregate_events(local_event=event, aggregated_events=AGGREGATED)
add_rows_to_create_event_table(AGGREGATED, event_table)
EventDisplayApp.run(
event_file=ARGS.falco_event,
title="Falco aggregated events report",
event_table=event_table
)
except KeyboardInterrupt:
CONSOLE.print("[bold]Program interrupted...[/bold]")
ファイルがディクショナリとしてロードされたら、イベントを集約するためにファイルを反復処理するだけで、結果をカウントでソートされたきちんとしたテーブルとして表示できます。
ファルコルールの表示方法
あなたが私のようなら、あなたは常に/etc/falco/falco_rules.yamlファイルを見て、何が監視されているかを理解しています。これらのルールの簡単なビュー（コメント付きの詳細なYAMLファイルを見ずに）は素晴らしい追加です：
#!/usr/bin/env python3
"""
Show brief content of default Falco rule YAML files
Jose Vicente Nunez (kodegeek.com@protonmail.com)
"""
from argparse import ArgumentParser
from pathlib import Path
from rich.console import Console
import yaml
from falcotutor.ui import create_rules_table, add_rows_to_create_rules_table, RulesDisplayApp
def load_rulez(falco_rulez: Path) -> dict[any, any]:
rulez = {}
with open(falco_rulez, 'rt') as falco_file:
for rule_data in yaml.full_load(falco_file):
if 'rule' in rule_data:
rule_name = rule_data['rule']
del rule_data['rule']
rulez[rule_name] = rule_data
return rulez
if __name__ == "__main__":
CONSOLE = Console()
AGGREGATED = {}
PARSER = ArgumentParser(description=__doc__)
PARSER.add_argument(
"falco_rules",
action="store"
)
ARGS = PARSER.parse_args()
try:
RULES = load_rulez(ARGS.falco_rules)
RULE_TBL = create_rules_table()
add_rows_to_create_rules_table(lrules=RULES, rules_tbl=RULE_TBL)
RulesDisplayApp.run(
rules_file=ARGS.falco_rules,
title="Falco brief rule display",
rules_table=RULE_TBL
)
except KeyboardInterrupt:
CONSOLE.print("[bold]Program interrupted...[/bold]")
たとえば、特定の基準（ルール名、優先度、有効/無効）によるルールフィルタリングを追加することで、このスクリプトを改善できます。このバージョンはフィルタリングを行いません。
Falco統合
以前の実験から、おそらく2つのことに気づいたでしょう。
- イベントのペイロードにはホストがありません。問題のあるサーバーを特定する場合は、マルチホストイベントの報告方法を改善する必要があります（多くのホストからのjournalctlファイルの解析は実用的ではありません）。
- 一元化された場所でアラートを受け取りたい。私たちが釣りに行く代わりに、それらのイベントを「プッシュ」する方法があるといいでしょう。
これらのアラートを1か所に統合するときが来ました。
FalcoExporterの使用方法
Falcoエクスポーターを使用すると、FalcoアラートをPrometheusスクレーパーと共有できます。最初に/etc/falco/falco.yamlでgRPCを有効にする必要があります
# gRPC server using an unix socket
grpc:
enabled: true
bind_address: "unix:///var/run/falco.sock"
# when threadiness is 0, Falco automatically guesses it depending on the number of online cores
threadiness: 0
# gRPC output service.
# By default it is off.
# By enabling this all the output events will be kept in memory until you read them with a gRPC client.
# Make sure to have a consumer for them or leave this disabled.
grpc_output:
enabled: true
Falcoを再起動します。
[root@macmini2 ~]# systemctl restart falco.service
[root@macmini2 ~]# systemctl status falco.service
● falco.service - Falco: Container Native Runtime Security
Loaded: loaded (/usr/lib/systemd/system/falco.service; disabled; vendor preset: disabled)
Active: active (running) since Sun 2022-05-01 20:35:01 EDT; 26s ago
Docs: https://falco.org/docs/
Process: 28285 ExecStartPre=/sbin/modprobe falco (code=exited, status=0/SUCCESS)
Main PID: 28288 (falco)
Tasks: 11 (limit: 2310)
Memory: 80.9M
CGroup: /system.slice/falco.service
└─28288 /usr/bin/falco --pidfile=/var/run/falco.pid
May 01 20:35:01 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 20:35:01 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 20:35:01 macmini2 falco[28288]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 20:35:01 macmini2 falco[28288]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 20:35:01 macmini2 falco[28288]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 20:35:02 macmini2 falco[28288]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 20:35:03 macmini2 falco[28288]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
May 01 20:35:04 macmini2 falco[28288]: Starting internal webserver, listening on port 8765
May 01 20:35:04 macmini2 falco[28288]: gRPC server threadiness equals to 2
May 01 20:35:04 macmini2 falco[28288]: Starting gRPC server at unix:///var/run/falco.sock
すべてが正常であることをすばやく確認します（リマインダー、Falcoエージェントはmacmini2で実行されています）：
josevnz@raspberrypi:~$ curl --fail http://macmini2:8765/healthz
{"status": "ok"}josevnz@raspberrypi:~$
次に、falco-exporterを実行します。簡単にするために、コマンドラインでいくつかのオーバーライドを使用してDockerコンテナを使用します。
[root@macmini2 ~]# docker run --restart always --name falco-exporter --detach --volume /var/run/falco.sock:/var/run/falco.sock --network=host falcosecurity/falco-exporter --listen-address 192.168.1.16:9376
7d157af0251ea4bc73b8c355a74eaf4dd24a5348cbe3f5f2ea9d7147c6c366c8
[root@macmini2 ~]# docker logs falco-exporter
2022/05/02 00:56:30 connecting to gRPC server at unix:///var/run/falco.sock (timeout 2m0s)
2022/05/02 00:56:30 listening on http://192.168.1.16:9376/metrics
2022/05/02 00:56:30 connected to gRPC server, subscribing events stream
2022/05/02 00:56:30 ready
# Check with CURL if the URL is reachable
[root@macmini2 ~]# curl http://192.168.1.16:9376/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 18
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.14.15"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.011112e+06
完全を期すために、ノードエクスポーターを使用してホストパフォーマンスメトリックをキャプチャする方法も示します（後でFalcoが使用するリソースの数を監視し、インストールによってサーバーに悪影響が及ばないようにするために使用します）。
docker run --detach --net="host" --pid="host" --volume "/:/host:ro,rslave" quay.io/prometheus/node-exporter:latest --path.rootfs=/host
node-exporterとfalco-exporterは、データのスクレイプが必要なすべてのホストで実行されます。次に、これらすべてのメトリックを1つの場所に収集するのを待つ必要があります。そのためには、Prometheusエージェントを使用します。
---
# /etc/prometheus.yaml on raspberrypi
global:
scrape_interval: 30s
evaluation_interval: 30s
scrape_timeout: 10s
external_labels:
monitor: 'nunez-family-monitor'
scrape_configs:
- job_name: 'falco-exporter'
static_configs:
- targets: ['macmini2.home:9376']
- job_name: 'node-exporter'
static_configs:
- targets: ['macmini2.home:9100', 'raspberrypi.home:9100', 'dmaf5:9100']
- job_name: 'docker-exporter'
static_configs:
- targets: ['macmini2.home:9323', 'raspberrypi.home:9323', 'dmaf5:9323']
tls_config:
insecure_skip_verify: true
次に、Prometheusスクレーパーが各ノードと通信できることを確認します。WebUIにアクセスします。
いいですね、プロメテウスはファルコをこすり落とすことができます。簡単なクエリを実行して、いくつかのイベントを確認することもできます。
次に、イベントのUIビューを設定する必要があります。そのために、Grafanaを使用します。
Grafanaをインストールする方法はたくさんあります。私の場合、Grafana Dockerコンテナーを使用します（Prometheusが実行されているのと同じホストでGrafanaを実行します：raspberripi.home）：
docker pull grafana/grafana:main-ubuntu
mkdir -p /data/grafana
chown syslog /data/grafana
docker run --user 104 --name grafana --detach --tty --volume /data/grafana:/var/lib/grafana -p 3000:3000 grafana/grafana:main-ubuntu
Grafanaが表示されたら、パスワードを変更する必要があります。また、Prometheusに接続する必要があります。
Grafanaが起動したら、ここで説明するようにFalcoダッシュボードをインポートできます。
ダッシュボードがインポートされると、インストールされているホストでFalcoをトリガーするいくつかのイベントを生成できます。
[root@macmini2 ~]# for i in $(seq 1 60); do docker run --rm --interactive --tty --privileged fedora:latest /bin/bash -c ls; touch /root/test; rm -f /root/test; sleep 1; done
しばらくすると、Grafanaダッシュボードに次のようなものが表示されます。
イベントは流れており、どのホストからイベントが来たかを確認できます。
Falcoイベントのアラートを作成する方法
理想的には、GrafanaでFalcoイベントがある場合は、これらの実用的なアイテムを作成して、そこからアラートを生成できます。
重要ではないアラートに襲われたくないので、最初に知っておくべきことは、フィルタリングするイベントのレベルです。
優先度が3未満の場合は、アラートとして扱われます。
Grafanaにはアラートの設定方法に関する優れたドキュメントがあるため、ここでは最終結果のみを示します。
次のステップは、アラートをどこかに送信することです。
アラートはどこかに行く必要がある–Discordを使用して連絡先を定義する方法
この例では、アラートの最後としてDiscordを使用します。Discordには、WebHookのセットアップ方法に関する非常に詳細なガイドがあるため、ここでは、DiscordWebhookの最終結果のみを示します。
そのURLをコピーしてから、Discord Webhookを使用する新しいGrafanaコンタクトポイントを構成します（これをすべてのアラートのデフォルトのコンタクトポイントに設定しています）。
そこから、このパイプラインが機能することを確認するために、Discordにテストメッセージを送信できます。
近づいています。ここまででアラート定義に戻ると、アラートが「発火」状態になっていることがわかります。
そして、すべてがうまくいけば、Discordで最初のFalcoアラートも表示されます。
ここで、journalctl出力で取得したすべてのフィールドを確認できます。違いは、これらのメッセージはすべて、Falco-Prometheus-Grafanaブリッジを定義するすべてのサーバーから送信されることです。
佳作：Falcon Sidekick /FalconSidekickを使用してアラートを集約する方法-UI
Falco Sidekickは、 Falco Sidekick-UIなど、イベントを収集して他の宛先に送信するもう1つの方法です。ただし、元のホストはわかりません（少なくともFalco 0.31.1まで）。
これは、K8sクラスターまたはコンテナー化されたアプリケーションからのアラートの問題ではない可能性が高く、イメージ名から多くの情報が得られます。ただし、イベントがベアメタル環境で発生し、2台以上のマシンがある場合、それは頭痛の種になります。
そのため、ここではSidekickについては説明しません。当面は、Grafanaの統合に固執することをお勧めします。
#falco #grafana #docker #cloud
What is GEEK
Buddha Community
1652299200
Falco、Prometheus、Grafana、Dockerを使用してサーバーインフラストラクチャクラウドを保護する
私は最近、コンテナとアプリケーションを機能させ続ける方法を探していました。具体的には、構成の異常を検出することに興味がありました。少し調べた後、私はFalcoに出くわしました。
私が見つけたのは、多くの機能と優れたドキュメントを備えた非常に完全なオープンソースプラットフォームでした。だから私はあなたと私の経験を共有したいと思いました。
この記事では何を取り上げますか？
- イベント（異常/違反）を監視するホストにFalcoエージェントをインストールする方法
- Falcoを調整して誤検知を減らし、本当に必要な情報を取得する方法
- Prometheusを使用して、輸出業者とスクレーパーの助けを借りて、Falcoイベントを中央の場所に収集する方法。
- 最後に、視覚化とアラートのためにスクレーパーをGrafanaに接続する方法
このチュートリアルには何が必要ですか？
- Linuxがインストールされている1つまたは複数のマシン。仮想マシンが機能するはずです。
- Docker、RPM、およびsystemdプロセスをインストール/セットアップできるようにするには、スーパーユーザー権限が必要です。
- Dockerコンテナーを使用するため、Dockerの基本的な知識が必要です。
- 構成をテストおよび改善するためのスクリプトをいくつか作成するため、Python/Bashの実用的な知識。
最後に、次の各コンポーネントをセットアップできるようになります。
恐れることはありません。ドキュメントへのリンクと、これらの各タスクの詳細な説明を提供します。
目次
ファルコとは？
このツールを説明する最良の方法は、ツールで何ができるかを学ぶことです。
Falcoは、Linuxシステムコールの実行に関連するすべての動作を検出して警告することができます。
Falcoアラートは、特定のシステムコールとその引数の使用、および呼び出しプロセスのプロパティによってトリガーできます。たとえば、Falcoは、以下を含むがこれらに限定されないインシデントを簡単に検出できます。
- Kubernetesのコンテナまたはポッド内でシェルが実行されています。
- コンテナが特権モードで実行されているか、ホストから/procなどの機密パスをマウントしています。
- サーバープロセスは、予期しないタイプの子プロセスを生成しています。
- / etc/shadowなどの機密ファイルの予期しない読み取り。
- デバイス以外のファイルが/devに書き込まれます。
- lsなどの標準のシステムバイナリは、アウトバウンドネットワーク接続を確立しています。
- 特権ポッドがKubernetesクラスターで開始されます。
Falcoのインストール方法
RPMを使用してFalcoをインストールします（apt-get、さらにはDockerコンテナーにも同様の手順があります）。私の場合、ネイティブインストールが最適であり、RPMによって非常に簡単に実行できると感じました。
[josevnz@macmini2 ~]$ sudo -i dnf install https://download.falco.org/packages/rpm/falco-0.31.1-x86_64.rpm
Last metadata expiration check: 2:53:53 ago on Sun 01 May 2022 04:13:09 PM EDT.
falco-0.31.1-x86_64.rpm 1.7 MB/s | 12 MB 00:07
Dependencies resolved.
==============================================================================================================================================================================================================================================================
Package Architecture Version Repository Size
==============================================================================================================================================================================================================================================================
Installing:
falco x86_64 0.31.1-1 @commandline 12 M
Installing dependencies:
dkms noarch 2.8.1-4.20200214git5ca628c.fc30 updates 78 k
elfutils-libelf-devel x86_64 0.179-2.fc30 updates 27 k
kernel-devel x86_64 5.6.13-100.fc30 updates 14 M
Transaction Summary
==============================================================================================================================================================================================================================================================
Install 4 Packages
Total size: 26 M
Total download size: 14 M
Installed size: 92 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): elfutils-libelf-devel-0.179-2.fc30.x86_64.rpm 253 kB/s | 27 kB 00:00
(2/3): dkms-2.8.1-4.20200214git5ca628c.fc30.noarch.rpm 342 kB/s | 78 kB 00:00
(3/3): kernel-devel-5.6.13-100.fc30.x86_64.rpm 1.9 MB/s | 14 MB 00:07
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 1.8 MB/s | 14 MB 00:07
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : kernel-devel-5.6.13-100.fc30.x86_64 1/4
Running scriptlet: kernel-devel-5.6.13-100.fc30.x86_64 1/4
Installing : elfutils-libelf-devel-0.179-2.fc30.x86_64 2/4
Installing : dkms-2.8.1-4.20200214git5ca628c.fc30.noarch 3/4
Running scriptlet: dkms-2.8.1-4.20200214git5ca628c.fc30.noarch 3/4
Running scriptlet: falco-0.31.1-1.x86_64 4/4
Installing : falco-0.31.1-1.x86_64 4/4
Running scriptlet: falco-0.31.1-1.x86_64
基本構成
非常に基本的な出力処理を行う場合を除いて、JSON出力を有効にします。
# Whether to output events in json or text
json_output: true
理由はすぐに明らかになります。
次に、Falcoエージェントを開始します。
[josevnz@macmini2 falco]$ sudo systemctl start falco.service
[josevnz@macmini2 falco]$ sudo systemctl status falco.service
● falco.service - Falco: Container Native Runtime Security
Loaded: loaded (/usr/lib/systemd/system/falco.service; disabled; vendor preset: disabled)
Active: active (running) since Sun 2022-05-01 19:20:52 EDT; 1s ago
Docs: https://falco.org/docs/
Process: 26887 ExecStartPre=/sbin/modprobe falco (code=exited, status=0/SUCCESS)
Main PID: 26888 (falco)
Tasks: 1 (limit: 2310)
Memory: 65.8M
CGroup: /system.slice/falco.service
└─26888 /usr/bin/falco --pidfile=/var/run/falco.pid
May 01 19:20:52 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 19:20:52 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 19:20:52 macmini2 falco[26888]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 19:20:52 macmini2 falco[26888]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 19:20:52 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 19:20:53 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 19:20:54 macmini2 falco[26888]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
デフォルト設定をテストする方法
構成によっては、Falcoを起動した直後にイベントが発生する場合と発生しない場合があります。
[josevnz@macmini2 falco]$ sudo journalctl --unit falco --follow
-- Logs begin at Tue 2021-05-25 00:15:22 EDT. --
May 01 19:20:52 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 19:20:52 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 19:20:52 macmini2 falco[26888]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 19:20:52 macmini2 falco[26888]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 19:20:52 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 19:20:53 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 19:20:54 macmini2 falco[26888]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
May 01 19:20:55 macmini2 falco[26888]: Starting internal webserver, listening on port 8765
心配しないでください。Falcoに警告とアラートを記録させるいくつかのコマンドを実行します。これがどのように機能するかを見てみましょう！
特権コンテナを実行する方法
特権コンテナの使用は悪い習慣と見なされているので、このイベントがFalcoによって検出されるかどうかを見てみましょう。
[josevnz@macmini2 ~]$ docker run --rm --interactive --tty --privileged --volume /etc/shadow:/mnt/shadow fedora:latest ls -l /mnt/shadow
----------. 1 root root 1198 Nov 21 20:51 /mnt/shadow
そして私たちのファルコログ？
May 01 19:29:32 macmini2 falco[26888]: {"output":"19:29:32.918828894: Informational Privileged container started (user=root user_loginuid=0 command=container:bfb9637a47a6 kind_lumiere (id=bfb9637a47a6) image=fedora:latest)","priority":"Informational","rule":"Launch Privileged Container","source":"syscall","tags":["cis","container","mitre_lateral_movement","mitre_privilege_escalation"],"time":"2022-05-01T23:29:32.918828894Z", "output_fields": {"container.id":"bfb9637a47a6","container.image.repository":"fedora","container.image.tag":"latest","container.name":"kind_lumiere","evt.time":1651447772918828894,"proc.cmdline":"container:bfb9637a47a6","user.loginuid":0,"user.name":"root"}}
情報イベントとして表示されます。間違いなく注目すべきものの1つです。コンテナ上のアプリケーションに昇格された特権が必要かどうかを自問してください。
また、各メッセージにタグが付いていることにお気づきかもしれません。「mitre_*」のものに注意してください。これらは、攻撃と軽減に関するMitreAttackナレッジベースに関連しています。うん、あなたはそれらを読むのに少し時間を費やすでしょう。
/rootディレクトリにファイルを作成する方法
この例は、コンテナ内のボリュームと組み合わせたrootユーザーを悪用する方法を示しています...
[josevnz@macmini2 ~]$ docker run --rm --interactive --tty --user root --volume /root:/mnt/ fedora:latest touch /mnt/test_file
[josevnz@macmini2 ~]$
ファルコ反応：
May 01 19:32:02 macmini2 falco[26888]: {"output":"19:32:02.434286167: Informational Container with sensitive mount started (user=root user_loginuid=0 command=container:ef061174c7ef distracted_lalande (id=ef061174c7ef) image=fedora:latest mounts=/root:/mnt::true:rprivate)","priority":"Informational","rule":"Launch Sensitive Mount Container","source":"syscall","tags":["cis","container","mitre_lateral_movement"],"time":"2022-05-01T23:32:02.434286167Z", "output_fields": {"container.id":"ef061174c7ef","container.image.repository":"fedora","container.image.tag":"latest","container.mounts":"/root:/mnt::true:rprivate","container.name":"distracted_lalande","evt.time":1651447922434286167,"proc.cmdline":"container:ef061174c7ef","user.loginuid":0,"user.name":"root"}}
敏感なマウントが検出されました！
/binにファイルを作成して賭け金を上げましょう
さて、これを行うとしましょう：
[josevnz@macmini2 ~]$ sudo -i
[root@macmini2 ~]# touch /bin/should_not_be_here
ファルコはそれについてどう思いますか？
May 01 19:36:41 macmini2 falco[26888]: {"output":"19:36:41.237634398: Error File below a known binary directory opened for writing (user=root user_loginuid=1000 command=touch /bin/should_not_be_here file=/bin/should_not_be_here parent=bash pcmdline=bash gparent=sudo container_id=host image=<NA>)","priority":"Error","rule":"Write below binary dir","source":"syscall","tags":["filesystem","mitre_persistence"],"time":"2022-05-01T23:36:41.237634398Z", "output_fields": {"container.id":"host","container.image.repository":null,"evt.time":1651448201237634398,"fd.name":"/bin/should_not_be_here","proc.aname[2]":"sudo","proc.cmdline":"touch /bin/should_not_be_here","proc.pcmdline":"bash","proc.pname":"bash","user.loginuid":1000,"user.name":"root"}}
エラー、書き込み用に開かれたバイナリディレクトリ。良いキャッチ。
デフォルトは必ずしも良いとは限りません
Falcoがしばらく実行された後、どのような種類のイベントを無視し、どのイベントを調査したいのかを把握することをお勧めします。
最初のステップは、ペイロードでJSON形式を使用して、すべてのイベントのリストを取得することです。
sudo journalctl --unit falco --no-page --output=cat > /tmp/falco_json_lines.txt
'output = cat'は、タイムスタンプなしでメッセージペイロードを提供するようにjournalctlに指示します（心配しないでください。JSONメッセージ自体にタイムスタンプがあります）。
Starting Falco: Container Native Runtime Security...
Started Falco: Container Native Runtime Security.
Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
Falco initialized with configuration file /etc/falco/falco.yaml
Loading rules from file /etc/falco/falco_rules.yaml:
Loading rules from file /etc/falco/falco_rules.local.yaml:
Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Starting internal webserver, listening on port 8765
{"output":"19:29:32.918828894: Informational Privileged container started (user=root user_loginuid=0 command=container:bfb9637a47a6 kind_lumiere (id=bfb9637a47a6) image=fedora:latest)","priority":"Informational","rule":"Launch Privileged Container","source":"syscall","tags":["cis","container","mitre_lateral_movement","mitre_privilege_escalation"],"time":"2022-05-01T23:29:32.918828894Z", "output_fields": {"container.id":"bfb9637a47a6","container.image.repository":"fedora","container.image.tag":"latest","container.name":"kind_lumiere","evt.time":1651447772918828894,"proc.cmdline":"container:bfb9637a47a6","user.loginuid":0,"user.name":"root"}}
{"output":"19:32:02.434286167: Informational Container with sensitive mount started (user=root user_loginuid=0 command=container:ef061174c7ef distracted_lalande (id=ef061174c7ef) image=fedora:latest mounts=/root:/mnt::true:rprivate)","priority":"Informational","rule":"Launch Sensitive Mount Container","source":"syscall","tags":["cis","container","mitre_lateral_movement"],"time":"2022-05-01T23:32:02.434286167Z", "output_fields": {"container.id":"ef061174c7ef","container.image.repository":"fedora","container.image.tag":"latest","container.mounts":"/root:/mnt::true:rprivate","container.name":"distracted_lalande","evt.time":1651447922434286167,"proc.cmdline":"container:ef061174c7ef","user.loginuid":0,"user.name":"root"}}
これまでのところ面白そうですが、これはどうですか？
{"output":"23:04:10.609949471: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=openat command=bash fd.name=/home/josevnz/.bash_history-01112.tmp name=/home/josevnz/.bash_history-01112.tmp path=<NA> oldpath=<NA> host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.609949471Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":"/home/josevnz/.bash_history-01112.tmp","evt.arg.oldpath":null,"evt.arg.path":null,"evt.time":1651633450609949471,"evt.type":"openat","fd.name":"/home/josevnz/.bash_history-01112.tmp","proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.635602857: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=openat command=bash fd.name=/home/josevnz/.bash_history-01627.tmp name=/home/josevnz/.bash_history-01627.tmp path=<NA> oldpath=<NA> host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.635602857Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":"/home/josevnz/.bash_history-01627.tmp","evt.arg.oldpath":null,"evt.arg.path":null,"evt.time":1651633450635602857,"evt.type":"openat","fd.name":"/home/josevnz/.bash_history-01627.tmp","proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.635851215: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=rename command=bash fd.name=<NA> name=<NA> path=<NA> oldpath=/home/josevnz/.bash_history-01627.tmp host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.635851215Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":null,"evt.arg.oldpath":"/home/josevnz/.bash_history-01627.tmp","evt.arg.path":null,"evt.time":1651633450635851215,"evt.type":"rename","fd.name":null,"proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.661829867: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=rename command=bash fd.name=<NA> name=<NA> path=<NA> oldpath=/home/josevnz/.bash_history-01112.tmp host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.661829867Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":null,"evt.arg.oldpath":"/home/josevnz/.bash_history-01112.tmp","evt.arg.path":null,"evt.time":1651633450661829867,"evt.type":"rename","fd.name":null,"proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
これは通常の/正当な操作です。このルールを強化するか、完全に削除する方法を見つけましょう。
まず、/etc/falco/falco_rules.yamlファイルを開き、ルール「シェル履歴の削除または名前変更」（前に見たJSON出力）を探します。
- list: docker_binaries
items: [docker, dockerd, exe, docker-compose, docker-entrypoi, docker-runc-cur, docker-current, dockerd-current]
macro: var_lib_docker_filepath
condition: (evt.arg.name startswith /var/lib/docker or fd.name startswith /var/lib/docker)
- rule: Delete or rename shell history
desc: Detect shell history deletion
condition: >
(modify_shell_history or truncate_shell_history) and
not var_lib_docker_filepath and
not proc.name in (docker_binaries)
output: >
Shell history had been deleted or renamed (user=%user.name user_loginuid=%user.loginuid type=%evt.type command=%proc.cmdline fd.name=%fd.name name=%evt.arg.name path=%evt.arg.path oldpath=%evt.arg.oldpath %container.info)
priority:
WARNING
tags: [process, mitre_defense_evasion]
Falcoのルールは、公式ドキュメントで詳細に説明されています。この作品を見るだけで、いくつかのことに気付くでしょう。
条件について：
- 複雑なロジックをサポートし、
- のようなマクロ
var_lib_docker_filepath - のようなリスト
(docker_binaries) - のようなフィールドを持つ特別な変数
proc.name。
このファイルは変更しないことをお勧めします。/etc/falco/falco_rules.local.yaml代わりに、 ：で必要なものをオーバーライドする必要があります。
# Add new rules, like this one
# - rule: The program "sudo" is run in a container
# desc: An event will trigger every time you run sudo in a container
# condition: evt.type = execve and evt.dir=< and container.id != host and proc.name = sudo
# output: "Sudo run in container (user=%user.name %container.info parent=%proc.pname cmdline=%proc.cmdline)"
# priority: ERROR
# tags: [users, container]
# Or override/append to any rule, macro, or list from the Default Rules
例として、スーパーユーザー（root）の履歴が上書きされる場合は注意しますが、それ以外の場合は問題ないとします。最良の部分は、ルール全体をオーバーライドする必要がないことです。
したがって、元のルールには条件が追加されます。
- rule: Delete or rename shell history
append: true
condition: and user.name=root
ルールが適切に記述されていることを検証することは常に良い考えです。そのためには、元のルールとオーバーライドを一緒にチェックするようにFalcoに指示する必要があります。
[root@macmini2 ~]# falco --validate /etc/falco/falco_rules.yaml --validate /etc/falco/falco_rules.local.yaml
Fri May 6 20:48:00 2022: Validating rules file(s):
Fri May 6 20:48:00 2022: /etc/falco/falco_rules.yaml
Fri May 6 20:48:00 2022: /etc/falco/falco_rules.local.yaml
/etc/falco/falco_rules.yaml: Ok
/etc/falco/falco_rules.local.yaml: Ok
Fri May 6 20:48:01 2022: Ok
# If the rules are OK, restart Falco
[root@macmini2 ~]# systemctl restart falco.service
Pythonで簡単なイベントエクスプローラーを作成する方法
どのルールがノイズであり、どのルールが有用であるかを理解するのは面倒であることにおそらく同意するでしょう。
このデータを正規化する必要があり、次のようなPythonスクリプトを使用します。
- 非JSONデータを削除する
- タイムスタンプなしでイベントタイプを集約します
- いくつかの集計統計を生成して、システムで最も頻繁に発生するイベントに集中できるようにします
小さなPythonスクリプトでうまくいくことができます。UIレンダリング部分を省略し（コードを確認して全体像を確認してください）、代わりにファイル解析ビットを表示します。
#!/usr/bin/env python3
"""
Aggregate Falco events to make it easier to override rules
Jose Vicente Nunez (kodegeek.com@protonmail.com)
"""
import json
import re
from argparse import ArgumentParser
from pathlib import Path
from rich.console import Console
from falcotutor.ui import EventDisplayApp, create_event_table, add_rows_to_create_event_table
def filter_events(journalctl_out: Path) -> dict[any, any]:
"""
:param journalctl_out:
:return:
"""
with open(journalctl_out, 'r') as journalctl_file:
for row in journalctl_file:
if re.search("^{", row):
data = json.loads(row)
if 'rule' in data and 'output_fields' in data:
yield data
def aggregate_events(local_event: dict[any, any], aggregated_events: dict[any, any]):
rule = local_event['rule']
if rule not in aggregated_events:
aggregated_events[rule] = {
'count': 0,
'priority': local_event['priority'],
'last_timestamp': "",
'last_fields': ""
}
aggregated_events[rule]['count'] += 1
aggregated_events[rule]['last_timestamp'] = local_event['time']
del local_event['output_fields']['evt.time']
aggregated_events[rule]['last_fields'] = json.dumps(local_event['output_fields'], indent=True)
if __name__ == "__main__":
CONSOLE = Console()
AGGREGATED = {}
PARSER = ArgumentParser(description=__doc__)
PARSER.add_argument(
"falco_event",
action="store"
)
ARGS = PARSER.parse_args()
try:
event_table = create_event_table()
for event in filter_events(ARGS.falco_event):
aggregate_events(local_event=event, aggregated_events=AGGREGATED)
add_rows_to_create_event_table(AGGREGATED, event_table)
EventDisplayApp.run(
event_file=ARGS.falco_event,
title="Falco aggregated events report",
event_table=event_table
)
except KeyboardInterrupt:
CONSOLE.print("[bold]Program interrupted...[/bold]")
ファイルがディクショナリとしてロードされたら、イベントを集約するためにファイルを反復処理するだけで、結果をカウントでソートされたきちんとしたテーブルとして表示できます。
ファルコルールの表示方法
あなたが私のようなら、あなたは常に/etc/falco/falco_rules.yamlファイルを見て、何が監視されているかを理解しています。これらのルールの簡単なビュー（コメント付きの詳細なYAMLファイルを見ずに）は素晴らしい追加です：
#!/usr/bin/env python3
"""
Show brief content of default Falco rule YAML files
Jose Vicente Nunez (kodegeek.com@protonmail.com)
"""
from argparse import ArgumentParser
from pathlib import Path
from rich.console import Console
import yaml
from falcotutor.ui import create_rules_table, add_rows_to_create_rules_table, RulesDisplayApp
def load_rulez(falco_rulez: Path) -> dict[any, any]:
rulez = {}
with open(falco_rulez, 'rt') as falco_file:
for rule_data in yaml.full_load(falco_file):
if 'rule' in rule_data:
rule_name = rule_data['rule']
del rule_data['rule']
rulez[rule_name] = rule_data
return rulez
if __name__ == "__main__":
CONSOLE = Console()
AGGREGATED = {}
PARSER = ArgumentParser(description=__doc__)
PARSER.add_argument(
"falco_rules",
action="store"
)
ARGS = PARSER.parse_args()
try:
RULES = load_rulez(ARGS.falco_rules)
RULE_TBL = create_rules_table()
add_rows_to_create_rules_table(lrules=RULES, rules_tbl=RULE_TBL)
RulesDisplayApp.run(
rules_file=ARGS.falco_rules,
title="Falco brief rule display",
rules_table=RULE_TBL
)
except KeyboardInterrupt:
CONSOLE.print("[bold]Program interrupted...[/bold]")
たとえば、特定の基準（ルール名、優先度、有効/無効）によるルールフィルタリングを追加することで、このスクリプトを改善できます。このバージョンはフィルタリングを行いません。
Falco統合
以前の実験から、おそらく2つのことに気づいたでしょう。
- イベントのペイロードにはホストがありません。問題のあるサーバーを特定する場合は、マルチホストイベントの報告方法を改善する必要があります（多くのホストからのjournalctlファイルの解析は実用的ではありません）。
- 一元化された場所でアラートを受け取りたい。私たちが釣りに行く代わりに、それらのイベントを「プッシュ」する方法があるといいでしょう。
これらのアラートを1か所に統合するときが来ました。
FalcoExporterの使用方法
Falcoエクスポーターを使用すると、FalcoアラートをPrometheusスクレーパーと共有できます。最初に/etc/falco/falco.yamlでgRPCを有効にする必要があります
# gRPC server using an unix socket
grpc:
enabled: true
bind_address: "unix:///var/run/falco.sock"
# when threadiness is 0, Falco automatically guesses it depending on the number of online cores
threadiness: 0
# gRPC output service.
# By default it is off.
# By enabling this all the output events will be kept in memory until you read them with a gRPC client.
# Make sure to have a consumer for them or leave this disabled.
grpc_output:
enabled: true
Falcoを再起動します。
[root@macmini2 ~]# systemctl restart falco.service
[root@macmini2 ~]# systemctl status falco.service
● falco.service - Falco: Container Native Runtime Security
Loaded: loaded (/usr/lib/systemd/system/falco.service; disabled; vendor preset: disabled)
Active: active (running) since Sun 2022-05-01 20:35:01 EDT; 26s ago
Docs: https://falco.org/docs/
Process: 28285 ExecStartPre=/sbin/modprobe falco (code=exited, status=0/SUCCESS)
Main PID: 28288 (falco)
Tasks: 11 (limit: 2310)
Memory: 80.9M
CGroup: /system.slice/falco.service
└─28288 /usr/bin/falco --pidfile=/var/run/falco.pid
May 01 20:35:01 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 20:35:01 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 20:35:01 macmini2 falco[28288]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 20:35:01 macmini2 falco[28288]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 20:35:01 macmini2 falco[28288]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 20:35:02 macmini2 falco[28288]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 20:35:03 macmini2 falco[28288]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
May 01 20:35:04 macmini2 falco[28288]: Starting internal webserver, listening on port 8765
May 01 20:35:04 macmini2 falco[28288]: gRPC server threadiness equals to 2
May 01 20:35:04 macmini2 falco[28288]: Starting gRPC server at unix:///var/run/falco.sock
すべてが正常であることをすばやく確認します（リマインダー、Falcoエージェントはmacmini2で実行されています）：
josevnz@raspberrypi:~$ curl --fail http://macmini2:8765/healthz
{"status": "ok"}josevnz@raspberrypi:~$
次に、falco-exporterを実行します。簡単にするために、コマンドラインでいくつかのオーバーライドを使用してDockerコンテナを使用します。
[root@macmini2 ~]# docker run --restart always --name falco-exporter --detach --volume /var/run/falco.sock:/var/run/falco.sock --network=host falcosecurity/falco-exporter --listen-address 192.168.1.16:9376
7d157af0251ea4bc73b8c355a74eaf4dd24a5348cbe3f5f2ea9d7147c6c366c8
[root@macmini2 ~]# docker logs falco-exporter
2022/05/02 00:56:30 connecting to gRPC server at unix:///var/run/falco.sock (timeout 2m0s)
2022/05/02 00:56:30 listening on http://192.168.1.16:9376/metrics
2022/05/02 00:56:30 connected to gRPC server, subscribing events stream
2022/05/02 00:56:30 ready
# Check with CURL if the URL is reachable
[root@macmini2 ~]# curl http://192.168.1.16:9376/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 18
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.14.15"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.011112e+06
完全を期すために、ノードエクスポーターを使用してホストパフォーマンスメトリックをキャプチャする方法も示します（後でFalcoが使用するリソースの数を監視し、インストールによってサーバーに悪影響が及ばないようにするために使用します）。
docker run --detach --net="host" --pid="host" --volume "/:/host:ro,rslave" quay.io/prometheus/node-exporter:latest --path.rootfs=/host
node-exporterとfalco-exporterは、データのスクレイプが必要なすべてのホストで実行されます。次に、これらすべてのメトリックを1つの場所に収集するのを待つ必要があります。そのためには、Prometheusエージェントを使用します。
---
# /etc/prometheus.yaml on raspberrypi
global:
scrape_interval: 30s
evaluation_interval: 30s
scrape_timeout: 10s
external_labels:
monitor: 'nunez-family-monitor'
scrape_configs:
- job_name: 'falco-exporter'
static_configs:
- targets: ['macmini2.home:9376']
- job_name: 'node-exporter'
static_configs:
- targets: ['macmini2.home:9100', 'raspberrypi.home:9100', 'dmaf5:9100']
- job_name: 'docker-exporter'
static_configs:
- targets: ['macmini2.home:9323', 'raspberrypi.home:9323', 'dmaf5:9323']
tls_config:
insecure_skip_verify: true
次に、Prometheusスクレーパーが各ノードと通信できることを確認します。WebUIにアクセスします。
いいですね、プロメテウスはファルコをこすり落とすことができます。簡単なクエリを実行して、いくつかのイベントを確認することもできます。
次に、イベントのUIビューを設定する必要があります。そのために、Grafanaを使用します。
Grafanaをインストールする方法はたくさんあります。私の場合、Grafana Dockerコンテナーを使用します（Prometheusが実行されているのと同じホストでGrafanaを実行します：raspberripi.home）：
docker pull grafana/grafana:main-ubuntu
mkdir -p /data/grafana
chown syslog /data/grafana
docker run --user 104 --name grafana --detach --tty --volume /data/grafana:/var/lib/grafana -p 3000:3000 grafana/grafana:main-ubuntu
Grafanaが表示されたら、パスワードを変更する必要があります。また、Prometheusに接続する必要があります。
Grafanaが起動したら、ここで説明するようにFalcoダッシュボードをインポートできます。
ダッシュボードがインポートされると、インストールされているホストでFalcoをトリガーするいくつかのイベントを生成できます。
[root@macmini2 ~]# for i in $(seq 1 60); do docker run --rm --interactive --tty --privileged fedora:latest /bin/bash -c ls; touch /root/test; rm -f /root/test; sleep 1; done
しばらくすると、Grafanaダッシュボードに次のようなものが表示されます。
イベントは流れており、どのホストからイベントが来たかを確認できます。
Falcoイベントのアラートを作成する方法
理想的には、GrafanaでFalcoイベントがある場合は、これらの実用的なアイテムを作成して、そこからアラートを生成できます。
重要ではないアラートに襲われたくないので、最初に知っておくべきことは、フィルタリングするイベントのレベルです。
優先度が3未満の場合は、アラートとして扱われます。
Grafanaにはアラートの設定方法に関する優れたドキュメントがあるため、ここでは最終結果のみを示します。
次のステップは、アラートをどこかに送信することです。
アラートはどこかに行く必要がある–Discordを使用して連絡先を定義する方法
この例では、アラートの最後としてDiscordを使用します。Discordには、WebHookのセットアップ方法に関する非常に詳細なガイドがあるため、ここでは、DiscordWebhookの最終結果のみを示します。
そのURLをコピーしてから、Discord Webhookを使用する新しいGrafanaコンタクトポイントを構成します（これをすべてのアラートのデフォルトのコンタクトポイントに設定しています）。
そこから、このパイプラインが機能することを確認するために、Discordにテストメッセージを送信できます。
近づいています。ここまででアラート定義に戻ると、アラートが「発火」状態になっていることがわかります。
そして、すべてがうまくいけば、Discordで最初のFalcoアラートも表示されます。
ここで、journalctl出力で取得したすべてのフィールドを確認できます。違いは、これらのメッセージはすべて、Falco-Prometheus-Grafanaブリッジを定義するすべてのサーバーから送信されることです。
佳作：Falcon Sidekick /FalconSidekickを使用してアラートを集約する方法-UI
Falco Sidekickは、 Falco Sidekick-UIなど、イベントを収集して他の宛先に送信するもう1つの方法です。ただし、元のホストはわかりません（少なくともFalco 0.31.1まで）。
これは、K8sクラスターまたはコンテナー化されたアプリケーションからのアラートの問題ではない可能性が高く、イメージ名から多くの情報が得られます。ただし、イベントがベアメタル環境で発生し、2台以上のマシンがある場合、それは頭痛の種になります。
そのため、ここではSidekickについては説明しません。当面は、Grafanaの統合に固執することをお勧めします。
1604034000
Monitoring YugabyteDB in Kubernetes with the Prometheus Operator and Grafana
This blog post is written by guest author Bhavin Gandhi, Software Engineer at InfraCloud Technologies, Inc. InfraCloud helps startups, SMBs, and enterprises scale by adopting cloud-native technologies.
Using the Prometheus Operator has become a common choice when it comes to running Prometheus in a Kubernetes cluster. It can manage Prometheus and Alertmanager for us with the help of CRDs in Kubernetes. The kube-prometheus-stack Helm chart (formerly known as prometheus-operator) comes with Grafana, node_exporter, and more out of the box.
In a previous blog post about Prometheus, we took a look at setting up Prometheus and Grafana using manifest files. We also explored a few of the metrics exposed by YugabyteDB. In this post, we will be setting up Prometheus and Grafana using the kube-prometheus-stack chart. And we will configure Prometheus to scrape YugabyteDB pods. At the end we will take a look at the YugabyteDB Grafana dashboard that can be used to visualize all the collected metrics.
Before we begin
Before we get started with the setup, make sure you have a Kubernetes 1.16+ cluster with kubectl pointing to it. You can create a GKE cluster or an equivalent in other cloud providers or use Minikube to create a local cluster.
We will be using Helm 3 to install the charts, make sure you have it installed on your machine as well.
Installing the Prometheus Operator
The kube-prometheus-stack Helm chart installs the required CRDs as well as the operator itself. The chart creates instances of Prometheus and Alertmanager, which are the custom resources managed by the operator.
It also comes with a set of components which one would expect to have while setting up monitoring in a Kubernetes cluster. These components include: node_exporter for collecting node level metrics, kube-state-metrics which exposes Kubernetes related metrics, and Grafana for visualizing the metrics collected by Prometheus. The chart also comes with default Grafana dashboards and Prometheus alerts.
To setup Helm with the prometheus-community and yugabytedb repositories, run the following command:
$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories
$ helm repo add yugabytedb https://charts.yugabyte.com
"yugabytedb" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "yugabytedb" chart repository
...Successfully got an update from the "prometheus-community" chart repository
Update Complete. ⎈ Happy Helming!⎈
To create a values file for kube-prometheus-stack, execute:
cat <<EOF > kube-prom-stack-values.yaml
grafana:
dashboardProviders:
dashboardproviders.yaml:
apiVersion: 1
providers:
- name: 'default'
orgId: 1
folder: ''
type: file
disableDeletion: false
editable: true
options:
path: /var/lib/grafana/dashboards/default
dashboards:
default:
yugabytedb:
url: https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/cloud/grafana/YugabyteDB.json
EOF
The above kube-prom-stack-values.yaml file tells Grafana to import the YugabyteDB dashboard from the given GitHub URL.
Now we will create a namespace and install kube-prometheus-stack in it.
$ kubectl create namespace monitoring
namespace/monitoring created
$ helm install prom prometheus-community/kube-prometheus-stack \
--namespace monitoring \
--values kube-prom-stack-values.yaml
To check if all the components are running fine, check the status of pods from the monitoring namespace.
$ kubectl get pods --namespace monitoring
NOTE: To keep things simple, we are not providing any other options to the Helm chart. Make sure you go through the README of kube-prometheus-stack and grafana chart to explore other options that you may need.
Installing and scraping YugabyteDB
With the monitoring setup done, let’s install the YugabyteDB Helm chart. It will install YugabyteDB and configure Prometheus to scrape (collect) the metrics from our pods. It uses the ServiceMonitor Custom Resource (CR) to achieve that.
To create a namespace and install the chart, run the following command:
$ kubectl create namespace yb-demo
namespace/yb-demo created
$ helm install cluster-1 yugabytedb/yugabyte \
--namespace yb-demo \
--set serviceMonitor.enabled=true \
--set serviceMonitor.extraLabels.release=prom
If you are using Minikube, refer to this documentation section and add the resource requirements accordingly.
To check the status of all the pods from the yb-demo namespace, execute the following command. Wait till all the pods have a Running status.
#databases #distributed sql #kubernetes #grafana #prometheus #prometheus operator
1652325960
How to Secure Server Infrastructure Clouds using Falco, Prometheus, Grafana and Docker
I was recently looking for a way to keep tabs on our containers and applications at work. Specifically, I was interested in detecting anomalies in the configuration. After a little research, I stumbled on Falco.
What I found was a very complete Open Source platform with lots of features and excellent documentation. So I wanted to share my experience with you.
What will we cover in this article?
- How to install the Falco agent on the host that you want to monitor for events (anomalies/violations)
- How to tune Falco to reduce false positives and get the information you really need
- How to use Prometheus to collect Falco events into a central location, with the help of the exporters and a scraper.
- Finally, how to connect the scraper with Grafana for visualization and alerting
What do you need for this tutorial?
- A machine or machines with Linux installed. A virtual machine should work.
- You will need superuser permissions to be able to install/setup Docker, RPM, and systemd processes
- We will use Docker containers, so basic knowledge of Docker is required
- Working knowledge of Python/Bash, as we will write a few scripts to test and improve our configuration.
At the end you will be able to setup each one of the following components:
Don't be intimidated – I'll provide links to the documentation and a thorough explanation of each one of these tasks as we move along.
Table of contents
- What is Falco?
- How to Install Falco
- Basic Configuration
- How to Test the Default Configuration
- Defaults Are Not Always Good
- Falco Integrations
- Learning More
What is Falco?
The best way to describe this tool is to learn what it can do:
Falco can detect and alert on any behavior that involves making Linux system calls.
Falco alerts can be triggered by the use of specific system calls, their arguments, and by properties of the calling process. For example, Falco can easily detect incidents including but not limited to:
- A shell is running inside a container or pod in Kubernetes.
- A container is running in privileged mode, or is mounting a sensitive path, such as /proc, from the host.
- A server process is spawning a child process of an unexpected type.
- Unexpected read of a sensitive file, such as /etc/shadow.
- A non-device file is written to /dev.
- A standard system binary, such as ls, is making an outbound network connection.
- A privileged pod is started in a Kubernetes cluster.
How to Install Falco
I will install Falco using an RPM (similar instructions exist for apt-get, and even Docker containers). In my case I felt the native installation was the best, and the RPM made it very easy to do:
[josevnz@macmini2 ~]$ sudo -i dnf install https://download.falco.org/packages/rpm/falco-0.31.1-x86_64.rpm
Last metadata expiration check: 2:53:53 ago on Sun 01 May 2022 04:13:09 PM EDT.
falco-0.31.1-x86_64.rpm 1.7 MB/s | 12 MB 00:07
Dependencies resolved.
==============================================================================================================================================================================================================================================================
Package Architecture Version Repository Size
==============================================================================================================================================================================================================================================================
Installing:
falco x86_64 0.31.1-1 @commandline 12 M
Installing dependencies:
dkms noarch 2.8.1-4.20200214git5ca628c.fc30 updates 78 k
elfutils-libelf-devel x86_64 0.179-2.fc30 updates 27 k
kernel-devel x86_64 5.6.13-100.fc30 updates 14 M
Transaction Summary
==============================================================================================================================================================================================================================================================
Install 4 Packages
Total size: 26 M
Total download size: 14 M
Installed size: 92 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): elfutils-libelf-devel-0.179-2.fc30.x86_64.rpm 253 kB/s | 27 kB 00:00
(2/3): dkms-2.8.1-4.20200214git5ca628c.fc30.noarch.rpm 342 kB/s | 78 kB 00:00
(3/3): kernel-devel-5.6.13-100.fc30.x86_64.rpm 1.9 MB/s | 14 MB 00:07
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 1.8 MB/s | 14 MB 00:07
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : kernel-devel-5.6.13-100.fc30.x86_64 1/4
Running scriptlet: kernel-devel-5.6.13-100.fc30.x86_64 1/4
Installing : elfutils-libelf-devel-0.179-2.fc30.x86_64 2/4
Installing : dkms-2.8.1-4.20200214git5ca628c.fc30.noarch 3/4
Running scriptlet: dkms-2.8.1-4.20200214git5ca628c.fc30.noarch 3/4
Running scriptlet: falco-0.31.1-1.x86_64 4/4
Installing : falco-0.31.1-1.x86_64 4/4
Running scriptlet: falco-0.31.1-1.x86_64
Basic Configuration
Unless we want to do very basic output processing, we want to enable the JSON output:
# Whether to output events in json or text
json_output: true
It will become evident why pretty soon.
Next start the Falco agent:
[josevnz@macmini2 falco]$ sudo systemctl start falco.service
[josevnz@macmini2 falco]$ sudo systemctl status falco.service
● falco.service - Falco: Container Native Runtime Security
Loaded: loaded (/usr/lib/systemd/system/falco.service; disabled; vendor preset: disabled)
Active: active (running) since Sun 2022-05-01 19:20:52 EDT; 1s ago
Docs: https://falco.org/docs/
Process: 26887 ExecStartPre=/sbin/modprobe falco (code=exited, status=0/SUCCESS)
Main PID: 26888 (falco)
Tasks: 1 (limit: 2310)
Memory: 65.8M
CGroup: /system.slice/falco.service
└─26888 /usr/bin/falco --pidfile=/var/run/falco.pid
May 01 19:20:52 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 19:20:52 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 19:20:52 macmini2 falco[26888]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 19:20:52 macmini2 falco[26888]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 19:20:52 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 19:20:53 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 19:20:54 macmini2 falco[26888]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
How to Test the Default Configuration
Depending on your configuration, you may or may not get any events right after starting Falco:
[josevnz@macmini2 falco]$ sudo journalctl --unit falco --follow
-- Logs begin at Tue 2021-05-25 00:15:22 EDT. --
May 01 19:20:52 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 19:20:52 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 19:20:52 macmini2 falco[26888]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 19:20:52 macmini2 falco[26888]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 19:20:52 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 19:20:53 macmini2 falco[26888]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 19:20:54 macmini2 falco[26888]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
May 01 19:20:55 macmini2 falco[26888]: Starting internal webserver, listening on port 8765
Worry not. We will run a few commands that will cause Falco to record some warning and alerts. Time to see how this works!
How to Run a Privileged Container
Using privileged containers is considered a bad practice, so let's see if this event is detected by Falco:
[josevnz@macmini2 ~]$ docker run --rm --interactive --tty --privileged --volume /etc/shadow:/mnt/shadow fedora:latest ls -l /mnt/shadow
----------. 1 root root 1198 Nov 21 20:51 /mnt/shadow
And our Falco log?
May 01 19:29:32 macmini2 falco[26888]: {"output":"19:29:32.918828894: Informational Privileged container started (user=root user_loginuid=0 command=container:bfb9637a47a6 kind_lumiere (id=bfb9637a47a6) image=fedora:latest)","priority":"Informational","rule":"Launch Privileged Container","source":"syscall","tags":["cis","container","mitre_lateral_movement","mitre_privilege_escalation"],"time":"2022-05-01T23:29:32.918828894Z", "output_fields": {"container.id":"bfb9637a47a6","container.image.repository":"fedora","container.image.tag":"latest","container.name":"kind_lumiere","evt.time":1651447772918828894,"proc.cmdline":"container:bfb9637a47a6","user.loginuid":0,"user.name":"root"}}
It shows up as an informational event. Definitely one of those things to keep an eye on. Ask yourself if the application on the container needs elevated privileges.
You also probably noticed that each message has tags. Pay attention to the "mitre_*" ones, they do relate to the Mitre Attack knowledge base of attacks and mitigations. Yep, you will spend some time reading those.
How to Create a File on the /root Directory
This example shows how to abuse the root user combined with volumes in a container...
[josevnz@macmini2 ~]$ docker run --rm --interactive --tty --user root --volume /root:/mnt/ fedora:latest touch /mnt/test_file
[josevnz@macmini2 ~]$
Falco reaction:
May 01 19:32:02 macmini2 falco[26888]: {"output":"19:32:02.434286167: Informational Container with sensitive mount started (user=root user_loginuid=0 command=container:ef061174c7ef distracted_lalande (id=ef061174c7ef) image=fedora:latest mounts=/root:/mnt::true:rprivate)","priority":"Informational","rule":"Launch Sensitive Mount Container","source":"syscall","tags":["cis","container","mitre_lateral_movement"],"time":"2022-05-01T23:32:02.434286167Z", "output_fields": {"container.id":"ef061174c7ef","container.image.repository":"fedora","container.image.tag":"latest","container.mounts":"/root:/mnt::true:rprivate","container.name":"distracted_lalande","evt.time":1651447922434286167,"proc.cmdline":"container:ef061174c7ef","user.loginuid":0,"user.name":"root"}}
Sensitive mount detected!
Let's Raise the Stakes by Creating a File on /bin
Alright let's say we do this:
[josevnz@macmini2 ~]$ sudo -i
[root@macmini2 ~]# touch /bin/should_not_be_here
What does Falco think about it?
May 01 19:36:41 macmini2 falco[26888]: {"output":"19:36:41.237634398: Error File below a known binary directory opened for writing (user=root user_loginuid=1000 command=touch /bin/should_not_be_here file=/bin/should_not_be_here parent=bash pcmdline=bash gparent=sudo container_id=host image=<NA>)","priority":"Error","rule":"Write below binary dir","source":"syscall","tags":["filesystem","mitre_persistence"],"time":"2022-05-01T23:36:41.237634398Z", "output_fields": {"container.id":"host","container.image.repository":null,"evt.time":1651448201237634398,"fd.name":"/bin/should_not_be_here","proc.aname[2]":"sudo","proc.cmdline":"touch /bin/should_not_be_here","proc.pcmdline":"bash","proc.pname":"bash","user.loginuid":1000,"user.name":"root"}}
An error, binary directory opened for writing. Good catch.
Defaults Are Not Always Good
After Falco is running for a while it is a good idea to get a sense of what kind of events we want to ignore and which ones we want to investigate.
The first step is to get a list of all the events, using our JSON format on the payload:
sudo journalctl --unit falco --no-page --output=cat > /tmp/falco_json_lines.txt
The 'output=cat' tells journalctl to give us the message payload without timestamps (don't worry, the JSON message itself has timestamps).
Starting Falco: Container Native Runtime Security...
Started Falco: Container Native Runtime Security.
Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
Falco initialized with configuration file /etc/falco/falco.yaml
Loading rules from file /etc/falco/falco_rules.yaml:
Loading rules from file /etc/falco/falco_rules.local.yaml:
Loading rules from file /etc/falco/k8s_audit_rules.yaml:
Starting internal webserver, listening on port 8765
{"output":"19:29:32.918828894: Informational Privileged container started (user=root user_loginuid=0 command=container:bfb9637a47a6 kind_lumiere (id=bfb9637a47a6) image=fedora:latest)","priority":"Informational","rule":"Launch Privileged Container","source":"syscall","tags":["cis","container","mitre_lateral_movement","mitre_privilege_escalation"],"time":"2022-05-01T23:29:32.918828894Z", "output_fields": {"container.id":"bfb9637a47a6","container.image.repository":"fedora","container.image.tag":"latest","container.name":"kind_lumiere","evt.time":1651447772918828894,"proc.cmdline":"container:bfb9637a47a6","user.loginuid":0,"user.name":"root"}}
{"output":"19:32:02.434286167: Informational Container with sensitive mount started (user=root user_loginuid=0 command=container:ef061174c7ef distracted_lalande (id=ef061174c7ef) image=fedora:latest mounts=/root:/mnt::true:rprivate)","priority":"Informational","rule":"Launch Sensitive Mount Container","source":"syscall","tags":["cis","container","mitre_lateral_movement"],"time":"2022-05-01T23:32:02.434286167Z", "output_fields": {"container.id":"ef061174c7ef","container.image.repository":"fedora","container.image.tag":"latest","container.mounts":"/root:/mnt::true:rprivate","container.name":"distracted_lalande","evt.time":1651447922434286167,"proc.cmdline":"container:ef061174c7ef","user.loginuid":0,"user.name":"root"}}
So far it looks interesting, but what about this?
{"output":"23:04:10.609949471: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=openat command=bash fd.name=/home/josevnz/.bash_history-01112.tmp name=/home/josevnz/.bash_history-01112.tmp path=<NA> oldpath=<NA> host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.609949471Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":"/home/josevnz/.bash_history-01112.tmp","evt.arg.oldpath":null,"evt.arg.path":null,"evt.time":1651633450609949471,"evt.type":"openat","fd.name":"/home/josevnz/.bash_history-01112.tmp","proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.635602857: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=openat command=bash fd.name=/home/josevnz/.bash_history-01627.tmp name=/home/josevnz/.bash_history-01627.tmp path=<NA> oldpath=<NA> host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.635602857Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":"/home/josevnz/.bash_history-01627.tmp","evt.arg.oldpath":null,"evt.arg.path":null,"evt.time":1651633450635602857,"evt.type":"openat","fd.name":"/home/josevnz/.bash_history-01627.tmp","proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.635851215: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=rename command=bash fd.name=<NA> name=<NA> path=<NA> oldpath=/home/josevnz/.bash_history-01627.tmp host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.635851215Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":null,"evt.arg.oldpath":"/home/josevnz/.bash_history-01627.tmp","evt.arg.path":null,"evt.time":1651633450635851215,"evt.type":"rename","fd.name":null,"proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
{"output":"23:04:10.661829867: Warning Shell history had been deleted or renamed (user=josevnz user_loginuid=1000 type=rename command=bash fd.name=<NA> name=<NA> path=<NA> oldpath=/home/josevnz/.bash_history-01112.tmp host (id=host))","priority":"Warning","rule":"Delete or rename shell history","source":"syscall","tags":["mitre_defense_evasion","process"],"time":"2022-05-04T03:04:10.661829867Z", "output_fields": {"container.id":"host","container.name":"host","evt.arg.name":null,"evt.arg.oldpath":"/home/josevnz/.bash_history-01112.tmp","evt.arg.path":null,"evt.time":1651633450661829867,"evt.type":"rename","fd.name":null,"proc.cmdline":"bash","user.loginuid":1000,"user.name":"josevnz"}}
This is a normal/ legitimate operation. Let's find a way to harden this rule or remove it completely.
First, open the /etc/falco/falco_rules.yaml file and look for the rule 'Delete or rename shell history' (JSON output we saw earlier):
- list: docker_binaries
items: [docker, dockerd, exe, docker-compose, docker-entrypoi, docker-runc-cur, docker-current, dockerd-current]
macro: var_lib_docker_filepath
condition: (evt.arg.name startswith /var/lib/docker or fd.name startswith /var/lib/docker)
- rule: Delete or rename shell history
desc: Detect shell history deletion
condition: >
(modify_shell_history or truncate_shell_history) and
not var_lib_docker_filepath and
not proc.name in (docker_binaries)
output: >
Shell history had been deleted or renamed (user=%user.name user_loginuid=%user.loginuid type=%evt.type command=%proc.cmdline fd.name=%fd.name name=%evt.arg.name path=%evt.arg.path oldpath=%evt.arg.oldpath %container.info)
priority:
WARNING
tags: [process, mitre_defense_evasion]
Falco rules are explained in detail on the official documentation. Just by looking at this piece you will notice a few things.
About the conditions:
- Support complex logic,
- macros like
var_lib_docker_filepath - lists like
(docker_binaries) - and special variables with fields like
proc.name.
It is recommended that do you not change this file. Instead you should override what you need on the /etc/falco/falco_rules.local.yaml:
# Add new rules, like this one
# - rule: The program "sudo" is run in a container
# desc: An event will trigger every time you run sudo in a container
# condition: evt.type = execve and evt.dir=< and container.id != host and proc.name = sudo
# output: "Sudo run in container (user=%user.name %container.info parent=%proc.pname cmdline=%proc.cmdline)"
# priority: ERROR
# tags: [users, container]
# Or override/append to any rule, macro, or list from the Default Rules
For the sake of example, say that we do care when the history of the super-user (root) is overridden, but everybody else if fine. The best part is that you don't have to override the whole rule.
So the original rule will get a condition appended:
- rule: Delete or rename shell history
append: true
condition: and user.name=root
It is always a good idea to validate that your rules are properly written. For that you need can tell Falco to check the original rules and your overrides together:
[root@macmini2 ~]# falco --validate /etc/falco/falco_rules.yaml --validate /etc/falco/falco_rules.local.yaml
Fri May 6 20:48:00 2022: Validating rules file(s):
Fri May 6 20:48:00 2022: /etc/falco/falco_rules.yaml
Fri May 6 20:48:00 2022: /etc/falco/falco_rules.local.yaml
/etc/falco/falco_rules.yaml: Ok
/etc/falco/falco_rules.local.yaml: Ok
Fri May 6 20:48:01 2022: Ok
# If the rules are OK, restart Falco
[root@macmini2 ~]# systemctl restart falco.service
How to Make a Simple Event Explorer in Python
You'll probably agree that getting a sense of what rules are noise and which ones are useful is tedious.
We need to normalize this data, and we will use a Python script that will:
- Remove non-JSON data
- Aggregate event types without the timestamps
- Generate a few aggregation statistics, so we can focus on the most frequent events in our system
A small Python script can do the trick. I'm leaving out the UI rendering part (please check the code to see the full picture), and instead will show you the file parsing bits:
#!/usr/bin/env python3
"""
Aggregate Falco events to make it easier to override rules
Jose Vicente Nunez (kodegeek.com@protonmail.com)
"""
import json
import re
from argparse import ArgumentParser
from pathlib import Path
from rich.console import Console
from falcotutor.ui import EventDisplayApp, create_event_table, add_rows_to_create_event_table
def filter_events(journalctl_out: Path) -> dict[any, any]:
"""
:param journalctl_out:
:return:
"""
with open(journalctl_out, 'r') as journalctl_file:
for row in journalctl_file:
if re.search("^{", row):
data = json.loads(row)
if 'rule' in data and 'output_fields' in data:
yield data
def aggregate_events(local_event: dict[any, any], aggregated_events: dict[any, any]):
rule = local_event['rule']
if rule not in aggregated_events:
aggregated_events[rule] = {
'count': 0,
'priority': local_event['priority'],
'last_timestamp': "",
'last_fields': ""
}
aggregated_events[rule]['count'] += 1
aggregated_events[rule]['last_timestamp'] = local_event['time']
del local_event['output_fields']['evt.time']
aggregated_events[rule]['last_fields'] = json.dumps(local_event['output_fields'], indent=True)
if __name__ == "__main__":
CONSOLE = Console()
AGGREGATED = {}
PARSER = ArgumentParser(description=__doc__)
PARSER.add_argument(
"falco_event",
action="store"
)
ARGS = PARSER.parse_args()
try:
event_table = create_event_table()
for event in filter_events(ARGS.falco_event):
aggregate_events(local_event=event, aggregated_events=AGGREGATED)
add_rows_to_create_event_table(AGGREGATED, event_table)
EventDisplayApp.run(
event_file=ARGS.falco_event,
title="Falco aggregated events report",
event_table=event_table
)
except KeyboardInterrupt:
CONSOLE.print("[bold]Program interrupted...[/bold]")
Once the file is loaded as a dictionary, we only need to iterate to it to aggregate the events, then show the results as a neat table sorted by count:
How to Show the Falco Rules
If you are like me, you are always looking at the /etc/falco/falco_rules.yaml file to understand what is being monitored. A brief view of those rules (without looking at the verbose YAML file with comments) is a nice addition:
#!/usr/bin/env python3
"""
Show brief content of default Falco rule YAML files
Jose Vicente Nunez (kodegeek.com@protonmail.com)
"""
from argparse import ArgumentParser
from pathlib import Path
from rich.console import Console
import yaml
from falcotutor.ui import create_rules_table, add_rows_to_create_rules_table, RulesDisplayApp
def load_rulez(falco_rulez: Path) -> dict[any, any]:
rulez = {}
with open(falco_rulez, 'rt') as falco_file:
for rule_data in yaml.full_load(falco_file):
if 'rule' in rule_data:
rule_name = rule_data['rule']
del rule_data['rule']
rulez[rule_name] = rule_data
return rulez
if __name__ == "__main__":
CONSOLE = Console()
AGGREGATED = {}
PARSER = ArgumentParser(description=__doc__)
PARSER.add_argument(
"falco_rules",
action="store"
)
ARGS = PARSER.parse_args()
try:
RULES = load_rulez(ARGS.falco_rules)
RULE_TBL = create_rules_table()
add_rows_to_create_rules_table(lrules=RULES, rules_tbl=RULE_TBL)
RulesDisplayApp.run(
rules_file=ARGS.falco_rules,
title="Falco brief rule display",
rules_table=RULE_TBL
)
except KeyboardInterrupt:
CONSOLE.print("[bold]Program interrupted...[/bold]")
You could improve this script by adding rule filtering by certain criteria, for example (rule name, priority, enabled/ disabled). This version doesn't do any filtering:
Falco Integrations
You probably noticed two things from our earlier experimentation:
- The payload of the events do not have the host. If you want to locate an offending server, you need to improve how a multi-host event is reported (parsing a journalctl file from many hosts is not practical).
- We want to get alerts in a centralized location. It would be nice to have a way to "push" those events instead of us going to fish.
It is time to consolidate those alerts in a single place.
How to Use Falco Exporter
The Falco exporter will allow us to share the Falco alerts with Prometheus scraper. We need first to enable gRPC in the /etc/falco/falco.yaml
# gRPC server using an unix socket
grpc:
enabled: true
bind_address: "unix:///var/run/falco.sock"
# when threadiness is 0, Falco automatically guesses it depending on the number of online cores
threadiness: 0
# gRPC output service.
# By default it is off.
# By enabling this all the output events will be kept in memory until you read them with a gRPC client.
# Make sure to have a consumer for them or leave this disabled.
grpc_output:
enabled: true
Restart Falco:
[root@macmini2 ~]# systemctl restart falco.service
[root@macmini2 ~]# systemctl status falco.service
● falco.service - Falco: Container Native Runtime Security
Loaded: loaded (/usr/lib/systemd/system/falco.service; disabled; vendor preset: disabled)
Active: active (running) since Sun 2022-05-01 20:35:01 EDT; 26s ago
Docs: https://falco.org/docs/
Process: 28285 ExecStartPre=/sbin/modprobe falco (code=exited, status=0/SUCCESS)
Main PID: 28288 (falco)
Tasks: 11 (limit: 2310)
Memory: 80.9M
CGroup: /system.slice/falco.service
└─28288 /usr/bin/falco --pidfile=/var/run/falco.pid
May 01 20:35:01 macmini2 systemd[1]: Starting Falco: Container Native Runtime Security...
May 01 20:35:01 macmini2 systemd[1]: Started Falco: Container Native Runtime Security.
May 01 20:35:01 macmini2 falco[28288]: Falco version 0.31.1 (driver version b7eb0dd65226a8dc254d228c8d950d07bf3521d2)
May 01 20:35:01 macmini2 falco[28288]: Falco initialized with configuration file /etc/falco/falco.yaml
May 01 20:35:01 macmini2 falco[28288]: Loading rules from file /etc/falco/falco_rules.yaml:
May 01 20:35:02 macmini2 falco[28288]: Loading rules from file /etc/falco/falco_rules.local.yaml:
May 01 20:35:03 macmini2 falco[28288]: Loading rules from file /etc/falco/k8s_audit_rules.yaml:
May 01 20:35:04 macmini2 falco[28288]: Starting internal webserver, listening on port 8765
May 01 20:35:04 macmini2 falco[28288]: gRPC server threadiness equals to 2
May 01 20:35:04 macmini2 falco[28288]: Starting gRPC server at unix:///var/run/falco.sock
Quickly make sure everything is OK (reminder, the Falco agent is running on macmini2):
josevnz@raspberrypi:~$ curl --fail http://macmini2:8765/healthz
{"status": "ok"}josevnz@raspberrypi:~$
Then we run the falco-exporter. To make it easier, we will use a Docker container with a few overrides in the command line.
[root@macmini2 ~]# docker run --restart always --name falco-exporter --detach --volume /var/run/falco.sock:/var/run/falco.sock --network=host falcosecurity/falco-exporter --listen-address 192.168.1.16:9376
7d157af0251ea4bc73b8c355a74eaf4dd24a5348cbe3f5f2ea9d7147c6c366c8
[root@macmini2 ~]# docker logs falco-exporter
2022/05/02 00:56:30 connecting to gRPC server at unix:///var/run/falco.sock (timeout 2m0s)
2022/05/02 00:56:30 listening on http://192.168.1.16:9376/metrics
2022/05/02 00:56:30 connected to gRPC server, subscribing events stream
2022/05/02 00:56:30 ready
# Check with CURL if the URL is reachable
[root@macmini2 ~]# curl http://192.168.1.16:9376/metrics
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 18
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.14.15"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.011112e+06
For completeness, let me show you also how to capture the host performance metrics using the node exporter (we will use it later to keep an eye on how many resources are used by Falco and to make sure our installation is not hurting the server):
docker run --detach --net="host" --pid="host" --volume "/:/host:ro,rslave" quay.io/prometheus/node-exporter:latest --path.rootfs=/host
The node-exporter and the falco-exporter will run on every host that needs their data scraped. Now you need to wait to collect all these metrics into a single location. For that will use the Prometheus agent:
---
# /etc/prometheus.yaml on raspberrypi
global:
scrape_interval: 30s
evaluation_interval: 30s
scrape_timeout: 10s
external_labels:
monitor: 'nunez-family-monitor'
scrape_configs:
- job_name: 'falco-exporter'
static_configs:
- targets: ['macmini2.home:9376']
- job_name: 'node-exporter'
static_configs:
- targets: ['macmini2.home:9100', 'raspberrypi.home:9100', 'dmaf5:9100']
- job_name: 'docker-exporter'
static_configs:
- targets: ['macmini2.home:9323', 'raspberrypi.home:9323', 'dmaf5:9323']
tls_config:
insecure_skip_verify: true
Then make sure the Prometheus scraper can talk with each one of the nodes. We visit the web UI:
Good, Prometheus is able to scrape Falco. We can even run a simple query to see a few events:
Next we need to setup the UI view for the events, and for that we will use Grafana.
There are many ways to install Grafana. In my case I will use a Grafana Docker container (I will run Grafana on the same host where Prometheus is running: raspberripi.home):
docker pull grafana/grafana:main-ubuntu
mkdir -p /data/grafana
chown syslog /data/grafana
docker run --user 104 --name grafana --detach --tty --volume /data/grafana:/var/lib/grafana -p 3000:3000 grafana/grafana:main-ubuntu
After Grafana comes up, you will need to change your password and will also need to connect with Prometheus:
Once Grafana is up, we can import the Falco dashboard as explained here.
Once the dashboard is imported we can generate a few events to trigger Falco on the host where is installed:
[root@macmini2 ~]# for i in $(seq 1 60); do docker run --rm --interactive --tty --privileged fedora:latest /bin/bash -c ls; touch /root/test; rm -f /root/test; sleep 1; done
After a little you should see something like this on your Grafana Dashboard:
The events are flowing, and you can see from which host they came.
How to Create Alerts for Your Falco Events
Ideally if you have the Falco events in Grafana, you can make these actionable items and generate alerts from then.
I don't want to get bombarded by non-critical alerts, so the first thing to know is to what level of events to filter:
Anything with priority below 3 will be treated as an alert.
Grafana has good documentation on how to setup an alert, so I will show here the end result only:
The next step is to send the alerts somewhere.
Alerts Need to Go Somewhere – How to Define a Contact Point using Discord
For this example we will use Discord as the end for the alerts. Discord has a very detailed guide on how to setup a WebHook, so I will only show you here the end results of my discord Webhook:
We copy that URL and then will configure a new Grafana contact point that uses our Discord webhook (we are setting this a default contact point for all the alerts):
From there we can send a test message to Discord, just to confirm that this pipeline works:
We're getting closer. By now if we go back to our alert definition we should see it is on the 'firing' state:
And if everything goes well we see also our first Falco alert in Discord:
We can see here all the fields we get on the journalctl output. The difference is that all these messages will come from all the servers where you define the Falco-Prometheus-Grafana bridge.
Honorable Mention: How to Aggregate Alerts using Falcon Sidekick/ Falcon Sidekick-UI
Falco Sidekick is another way to gather and send events to other destinations, like the Falco Sidekick-UI. But it won't tell you the originating host (at least until Falco 0.31.1).
This is most likely not an issue for an alert coming from a K8s cluster or a containerized application where the image name will give you plenty of information. But if your event happens on a bare-metal environment, and you have more than 2 machines, it will become a headache.
For that reason I won't cover Sidekick here – you may want to stick with the Grafana integration for the time being.
1621253580
Grafana dashboard tutorial: custom data visualization with Grafana + Prometheus
You may already know how important it is to measure the performance of your app, or even how to integrate with tools such as Prometheus. But if you can’t visualize your data in an easy-to-read and organized manner, you won’t get much out of it. There’s a cool, open-source tool for that. In the last part of the Metrics in optimization process series, we’re teaching you how to create a dashboard in Grafana, so that you can produce a new dashboard that even the business department will follow.
What you will learn here
In the previous three articles, we talked about why you should measure your app’s metrics, how you can integrate it with Prometheus and how to go about collecting statistics.
Since we have now a lot of data to show we should think about the presentation layer. The true purpose of it is to present data in a way that gives us what we need, rather than just as a disorganized pile of trash. It will not just make pretties, but also far more useful. Another important thing is that the data, which we collected with Prometheus, doesn’t have an easy-to-read format. Today it’s time to show ugly metrics data in a beautiful way.
As you know from previous articles, we collect data from Kubernetes and our application. We are going to show an example of how to configure a dashboard, what is important, and why we did it this way. We are not going to show and describe all options in Grafana, just the ones that we needed to configure our charts (hey, it’s a crash course, after all!).
#developer stories #data #devops #grafana #kubernetes #prometheus
1591101480
Getting Started with Prometheus and Grafana
This post is broken up into 3 separate sections : “Introduction”, “Setting up Prometheus” and “Setting up Grafana For Prometheus”. The First section will concern the explanation of System Monitoring, Prometheus, and its different components. The second section will deal with the installation and the setup up part of Prometheus on Ubuntu 18.04. The last section will deal with the configuration and the integration of Grafana to Prometheus
#grafana #devops #ubuntu #monitoring #prometheus