Monitoring your EKS clusters audit logs
This blog post is an update of a post of November 2022
At the beginning of the year 2022, Falco introduced a game changing feature, the Falco Plugins. They allow Falco to monitor and trigger alerts for any kind of event. Since the launch of the new plugin framework the Falco community has collaborated to create plugins for Github, AWS CloudTrail and Okta. A plugin has also replaced the way Falco consumes the Audit Logs generated by a K8s API server through a dedicated plugin. With these plugins, Falco covers more in depth the aspects of your infrastructure and allows you to use a single syntax for rules.
For months (okay, maybe years...), our adopters have asked us for a way to monitor K8s Audit Logs. The previous implementation used an internal web server to receive the logs from the Kubernetes API, although it was functional, it was a very manual process to install and manage clusters. This method didn't support clusters managed by cloud providers, such as EKS, AKS, or GKE as they had to capture the Audit Logs for their own usage and then add them to their log aggregators.
This situation is now solved thanks to the plugin framework and we're proud to announce the first release of the plugin for EKS Audit Logs: k8saudit-eks!!!
How it works
AWS captures the Audit Logs and exposes them in the CloudWatch Logs service. We have made available libs to create a clean session with the AWS API and pull logs from the relevant Cloudwatch Logs Stream. You can reuse these libs for any plugin you'd like to create for any Amazon service.
Usage
The configuration for the usage of the plugin is:
plugins:
- name: k8saudit-eks
library_path: libk8saudit-eks.so
init_config:
region: "{REGION}"
profile: "{AWS_PROFILE}"
shift: 10
polling_interval: 10
use_async: false
buffer_size: 500
open_params: "{CLUSTER_NAME}"
- name: json
library_path: libjson.so
init_config: ""
load_plugins: [k8saudit-eks, json]
With:
init_config:- profile: The Profile to use to create the session, env var AWS_PROFILE if present
- region: The Region of your EKS cluster, env var AWS_REGION is used if present
- use_async: If true then async extraction optimization is enabled (Default: true)
- polling_interval: Polling Interval in seconds (default: 5s)
- shift: Time shift in past in seconds (default: 1s)
- buffer_size: Buffer Size (default: 200)
open_params: A string which contains the name of your EKS Cluster (required).
If you run Falco inside an EKS cluster with a setup of an OIDC provider, the profile and region parameters can be omitted in favor of a service account + IAM Role (see the official docs).
IAM
Whatever the method you use for the authentication to the AWS API, you need to set up this minimal policy to your user/role:
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"ReadAccessToCloudWatchLogs",
"Effect":"Allow",
"Action":[
"logs:Describe*",
"logs:FilterLogEvents",
"logs:Get*",
"logs:List*"
],
"Resource":[
"arn:aws:logs:${REGION}:${ACCOUNT_ID}:log-group:/aws/eks/${CLUSTER_NAME}/cluster:*"
]
}
]
}
With:
REGION: The Region of your EKS clusterACCOUNT_ID: The ID of the account running the EKS clusterCLUSTER_NAME: The name of your cluster EKS cluster (same value than in your plugin configuration)
Default Rules
A good thing about Kubernetes is that it brings standards into our industry. Despite a few differences, the cluster works in the same way and produces the same format of logs. This helps us enormously. By creating the k8saudit plugin we declared the fields to extract, as well as some default rules, which we can reuse for any plugin that consumes the same Audit Logs. It is a time saver for both, developers and adopters.
You can find the proposed default rules here.
Installation
For the installation, we'll use falcoctl. We can use it as a CLI tool for a locale installation or as a sidecar for Kubernetes. See this blog post for more information.
Local installation
In this example, we'll see how to install the k8saudit-eks in a local host.
First, add the index of artifacts for falcoctl:
sudo falcoctl index add falcosecurity https://falcosecurity.github.io/falcoctl/index.yaml
Install the latest version of the plugins:
sudo falcoctl artifact install k8saudit-eks
sudo falcoctl artifact install json
As the output notices, the plugins will be now available in /usr/share/plugins:
INFO Reading all configured index files from "/root/.config/falcoctl/indexes.yaml"
INFO Resolving dependencies ...
INFO Installing the following artifacts: [ghcr.io/falcosecurity/plugins/plugin/k8saudit-eks:latest]
INFO Preparing to pull "ghcr.io/falcosecurity/plugins/plugin/k8saudit-eks:latest"
INFO Pulling b2daf90a878e: ############################################# 100%
INFO Pulling 60b0846d1e18: ############################################# 100%
INFO Pulling 12c7f0f5f00e: ############################################# 100%
INFO Artifact successfully installed in "/usr/share/plugins"
INFO Reading all configured index files from "/root/.config/falcoctl/indexes.yaml"
INFO Resolving dependencies ...
INFO Installing the following artifacts: [ghcr.io/falcosecurity/plugins/plugin/json:latest]
INFO Preparing to pull "ghcr.io/falcosecurity/plugins/plugin/json:latest"
INFO Pulling 087dba2e76d6: ############################################# 100%
INFO Pulling c4cb35bb528a: ############################################# 100%
INFO Pulling 9b50ea237bc6: ############################################# 100%
INFO Artifact successfully installed in "/usr/share/plugins"
After installing the k8saudit-eks plugin, we also need to install the rules that it uses. (Those rules are the same as the k8saudit plugin rules.):
sudo falcoctl artifact install k8saudit-rules --resolve-deps=false
And we get the rules in /etc/falco:
INFO Reading all configured index files from "/root/.config/falcoctl/indexes.yaml"
INFO Installing the following artifacts: [ghcr.io/falcosecurity/plugins/ruleset/k8saudit:latest]
INFO Preparing to pull "ghcr.io/falcosecurity/plugins/ruleset/k8saudit:latest"
INFO Pulling b8c6b1def3eb: ############################################# 100%
INFO Pulling 6b6f3231f2b2: ############################################# 100%
INFO Pulling 91b2b7a9944d: ############################################# 100%
INFO Artifact successfully installed in "/etc/falco"
By default, the installation of a ruleset comes with the associated plugins. In this blog, we have installed them on our own, this is why --resolve-deps is set to false.
We now have the plugins and rules in our system. The last step is to configure Falco to use them.
Modify your /etc/yaml to make it looks like the following (with the relevant values for REGION, ACCOUNT_ID, and CLUSTER_NAME):
rules_file:
...
- /etc/k8s_audit_rules.yaml
plugins:
- name: k8saudit-eks
library_path: libk8saudit-eks.so
init_config:
region: "{REGION}"
profile: "{AWS_PROFILE}"
shift: 10
polling_interval: 10
use_async: false
buffer_size: 500
open_params: "{CLUSTER_NAME}"
- name: json
library_path: libjson.so
init_config: ""
load_plugins: ["k8saudit-eks","json"]
...
You can now start Falco ( sudo /usr/bin/falco --disable-source syscall) and monitor the audit logs of your EKS cluster:
Wed May 10 14:07:18 2023: Falco version: 0.34.1 (x86_64)
Wed May 10 14:07:18 2023: Falco initialized with configuration file: /etc/yaml
Wed May 10 14:07:18 2023: Loading plugin 'k8saudit-eks' from file /usr/share/plugins/libk8saudit-eks.so
Wed May 10 14:07:18 2023: Loading plugin 'json' from file /usr/share/plugins/libjson.so
Wed May 10 14:07:18 2023: Loading rules from file /etc/k8s_audit_rules.yaml
Wed May 10 14:07:18 2023: Loading rules from file /etc/rules.d/override-k8saudit.yaml
Wed May 10 14:07:18 2023: Starting health webserver with threadiness 4, listening on port 8765
Wed May 10 14:07:18 2023: Enabled event sources: k8s_audit
Wed May 10 14:07:18 2023: Opening capture with plugin 'k8saudit-eks'
In this example, we also disable the collection of the syscalls with the option --disable-source syscall. The k8saudit-eks plugin works with a pull model, you should have only one Falco instance collecting the audit logs to avoid any duplication of alerts. We dedicate that instance to that role.
And get alerts:
14:09:17.699479000: Informational K8s Serviceaccount Created (user=system:node:ip-192-168-34-135.ec2.internal serviceaccount=default ns=default resource=serviceaccounts resp=201 decision=allow reason=)
14:09:24.046826000: Notice Attach/Exec to pod (user=kubernetes-admin pod=kubecon resource=pods ns=default action=exec command=sh)
Kubernetes installation
Next, we will see how to install the k8saudit-eks plugin with the falcoctl tool in a Kubernetes setup. We'll use the official Helm chart with the following values.yaml file:
serviceAccount:
create: true
annotations:
- eks.amazonaws.com/role-arn: {ROLE_ARN}
falco:
rules_file:
- /etc/falco/k8s_audit_rules.yaml
- /etc/falco/rules.d
plugins:
- name: k8saudit-eks
library_path: libk8saudit-eks.so
init_config:
shift: 10
polling_interval: 10
use_async: false
buffer_size: 500
open_params: "{CLUSTER_NAME}"
- name: json
library_path: libjson.so
init_config: ""
load_plugins: [k8saudit-eks, json]
falcosidekick:
enabled: true
driver:
enabled: false
collectors:
enabled: false
controller:
kind: deployment
deployment:
replicas: 1
falcoctl:
indexes:
- name: falcosecurity
url: https://falcosecurity.github.io/falcoctl/index.yaml
artifact:
install:
enabled: true
follow:
enabled: true
config:
artifact:
allowedTypes:
- plugin
- rulesfile
install:
resolveDeps: false
refs: [k8saudit-rules:0, k8saudit-eks:0, json:0]
follow:
refs: [k8saudit-rules:0]
Let's look into this configuration section by section:
serviceAccount:
create: true
annotations:
- eks.amazonaws.com/role-arn: {ROLE_ARN}
With EKS, we can use an OIDC provider and annotations on the service accounts to allow access to the AWS API ( see the official docs). This section allows you to set which ROLE_ARN to use.
falco:
rules_file:
- /etc/falco/k8s_audit_rules.yaml
- /etc/falco/rules.d
plugins:
- name: k8saudit-eks
library_path: libk8saudit-eks.so
init_config:
shift: 10
polling_interval: 10
use_async: false
buffer_size: 500
open_params: "{CLUSTER_NAME}"
- name: json
library_path: libjson.so
init_config: ""
load_plugins: [k8saudit-eks, json]
As the local installation, this section sets the used plugins.
driver:
enabled: false
collectors:
enabled: false
controller:
kind: deployment
deployment:
replicas: 1
To collect and monitor all the syscalls, Falco must be deployed as a DaemontSet in your cluster, to have one instance per Kernel. In this example, we're deploying Falco with the k8saudit-eks plugin, which relies on a pull model. To avoid several Falco pods to collect the same audit logs and duplicate the alerts, we MUST install Falco with the k8saudit-eks plugin only once. The Helm chart allows to use a one replica deployment and to disable the syscalls and K8s metdadata collections, this is the purpose of this section.
falcoctl:
indexes:
- name: falcosecurity
url: https://falcosecurity.github.io/falcoctl/index.yaml
artifact:
install:
enabled: true
follow:
enabled: true
config:
artifact:
allowedTypes:
- plugin
- rulesfile
install:
resolveDeps: false
refs: [k8saudit-rules:0, k8saudit-eks:0, json:0]
follow:
refs: [k8saudit-rules:0]
The last section sets up falcoctl. It will install the requested plugins and rules and track new versions of the rules.
Now you can run your helm install command:
helm install falco-k8saudit-eks -n falco falcosecurity/falco -f ./values.yml
And see the result:
$ kubectl get pods -n falco
NAME READY STATUS RESTARTS AGE
falco-k8saudit-eks-96d7f6f99-vwjgc 2/2 Running 0 70m
$ kubectl logs falco-k8saudit-eks-96d7f6f99-vwjgc -n falco -c falco
Wed May 10 14:07:18 2023: Falco version: 0.34.1 (x86_64)
Wed May 10 14:07:18 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Wed May 10 14:07:18 2023: Loading plugin 'k8saudit-eks' from file /usr/share/falco/plugins/libk8saudit-eks.so
Wed May 10 14:07:18 2023: Loading plugin 'json' from file /usr/share/falco/plugins/libjson.so
Wed May 10 14:07:18 2023: Loading rules from file /etc/falco/k8s_audit_rules.yaml
Wed May 10 14:07:18 2023: Loading rules from file /etc/falco/rules.d/override-k8saudit.yaml
Wed May 10 14:07:18 2023: Starting health webserver with threadiness 4, listening on port 8765
Wed May 10 14:07:18 2023: Enabled event sources: k8s_audit
Wed May 10 14:07:18 2023: Opening capture with plugin 'k8saudit-eks'
{"hostname":"falco-k8saudit-eks-96d7f6f99-vwjgc","output":"14:09:17.699479000: Informational K8s Serviceaccount Created (user=system:node:ip-192-168-34-135.ec2.internal serviceaccount=default ns=default resource=serviceaccounts resp=201 decision=allow reason=)","priority":"Informational","rule":"K8s Serviceaccount Created","source":"k8s_audit","tags":["k8s"],"time":"2023-05-10T14:09:17.699479000Z", "output_fields": {"evt.time":1683727757699479000,"ka.auth.decision":"allow","ka.auth.reason":"","ka.response.code":"201","ka.target.name":"default","ka.target.namespace":"default","ka.target.resource":"serviceaccounts","ka.user.name":"system:node:ip-192-168-34-135.ec2.internal"}}
{"hostname":"falco-k8saudit-eks-96d7f6f99-vwjgc","output":"14:09:24.046826000: Notice Attach/Exec to pod (user=kubernetes-admin pod=kubecon resource=pods ns=default action=exec command=sh)","priority":"Notice","rule":"Attach/Exec Pod","source":"k8s_audit","tags":["k8s"],"time":"2023-05-10T14:09:24.046826000Z", "output_fields": {"evt.time":1683727764046826000,"ka.target.name":"kubecon","ka.target.namespace":"default","ka.target.resource":"pods","ka.target.subresource":"exec","ka.uri.param[command]":"sh","ka.user.name":"kubernetes-admin"}}
Disclaimer
Our tests noticed some latencies between the presence of the logs in the CloudWatch Logs Stream and their evaluation by Falco. This is more visible with highly requested API servers. The solution is to adapt the size of your nodes where Falco runs, considering a minimal size of 2xlarge as a safe option.
Conclusion
With this first Plugin for a managed K8s solution, we hope to open the door for new contributions from the community for other flavors like GKE and AKS. If you need to create a plugin for another AWS service, take also a look at the libs we created to help the developers.
You can find us in the Falco community. Please feel free to reach out to us for any questions, suggestions, or even for a friendly chat!
If you would like to find out more about Falco:
- Get started in Falco.org
- Check out the Falco project in GitHub.
- Get involved in the Falco community.
- Meet the maintainers on the Falco Slack.
- Follow @falco_org on Twitter.