Blog

KubeArmor

How to Troubleshoot KubeArmor policies?

How to Troubleshoot KubeArmor policies?


In this blog, we are going to see how to troubleshoot KubeArmor policies in GKE.

This blog assumes that you have read the concepts which explain all the components and fundamentals of KubeArmor. The blog will cover the following key steps:

  • Creating policies
  • Applying policies
  • Testing policies
  • Troubleshooting policies

Creating policies

  1. To deploy KubeArmor in GKE apply this script. https://raw.githubusercontent.com/kubearmor/KubeArmor/master/deployments/GKE/kubearmor.yaml
  2. Another method is to deploy kubearmor in GKE follow this guide to deploy kubearmor in a very simple way.
  3. Sample PostgreSQL policy is given below:

    Sample PostgreSQL policy

    3. The above policy has been created for the PostgreSQL workload. Psql is the binary for PostgreSQL. If anyone tries to access the psql the policy will deny the access. Those binaries will be found easily in the bin folder inside the Linux directory.

    4. Let’s apply the policy and see it in action!

    Applying Policies

    1. When applying policies make sure to verify with YAML lint applications to verify the structure and indentation of the YAML files. If not verified, the policy will fail to apply.

    Note: The policy must follow the required policy structure of KubeArmor. View here


    KubeArmor structure

    2. The above error shows that you may have missed the YAML indentation. As shown below.

3. The above error can simply be fixed by changing the indentation. Make sure to use appropriate YAML lint extensions in your code editor to avoid these kinds of errors.

4. Now we have fixed the policy. Let’s try to apply that policy and see it in action.

[root@karmor]$ kubectl apply -f postgresql-policy.yaml kubearmorpolicy.security.kubearmor.com/ksp-stigs-postgresql-console created

5. Now our policy has been created and applied in the GKE.

6. Check here for a complete guide and specification to write a kubearmor policy.

Testing Policies

  1. When you are testing policies you have to check with KubeArmor Security policy specifications. https://github.com/kubearmor/KubeArmor/blob/main/getting-started/security_policy_specification.md
  2. Make sure the wordings are spelled correctly. Otherwise, it throws an error like this shown below. These types of errors are likely to happen when we misspell words. So make sure to check in with the contribution guide mentioned above.

Testing policy

3. The above error shows we misspelled the words “matchPath” instead of “matchPaths” and followed by “matchLabel” instead of “matchLabels” also “tag” instead of “tags”. Let’s fix those errors and test the policy again.

4. This is the fixed policy with correct spellings.

Kubearmor policy

Code block

5. Now our policy is applied and ready.

[root@kubearmor]$ kubectl apply -f postgres-policy.yaml kubearmorpolicy.security.kubearmor.com/ksp-stigs-postgresql created

6. Let’s deploy PostgreSQL deployment in our cluster to test the policy.

7. To deploy a Postgres service in your cluster we have to apply this script

Script-1

Script-2

Script-3

8. Now we have our working Postgres pod running in our k8s cluster.

Postgres pod

9. Now let’s try to access the psql binary to test if it’s working or not. The goal is to block access to the psql binary.

psql binary

10. As you can see in this above problem, psql binary is still accessible even after applying the policy. That policy is not working as we expected. Let’s troubleshoot where it went wrong and fix the policy.

Troubleshooting Policies

  1. Even after using the correct format and parameters, the policy is not working and psql binary is still accessible. Why?
  2. Let’s check the binary location with commands we know such as “whereis” and “which”. So we can figure out whether we added the correct path or not.
  3. Here we checked psql binary location “whereis” command It shows that we added the correct path in our policy but still the policy is not working as we expected,

    4. Let’s dive in deep and find if we have any other connections to the psql binary.    If there is, we can add that to our policy.

    Kubearmor pod

    5. Now when we check the psql with “ls -la” command. That psql binary is linked with "../share/postgresql-common/pg_wrapper”

    6. So now we can see that it’s linked with another binary. So that’s why it’s not blocking the psql binary. We can add that in our policy and test that again.

    7. Here’s the updated policy with that path included

Kubearmor policy

8. Let’s apply this policy and check whether it is blocking the psql binary or not.

9. Now our policy is created without any errors.

Policy template

11.  Now we have our working policy that can block the psql binary in postgres pod. So we can authorize the use of the postgres server.

KubeArmor Logs

  1. To check kubearmor generated logs. We have to type this command to see that.

2. kubectl -n kube-system get pods -A | grep kubearmor

3. This command will list all kubearmor pods as shown below.

Kubearmor pods

4. You have to select the first three pods that are named “kubearmor-1234” similarly. This name will differ in your environment. You have to enter the command shown below.

5. kubectl -n kube-system exec -it <kube-armor-pod> -- tail /tmp/kubearmor.log

6. You will see the logs like this in the terminal as shown in the image below

Kubearmor pod

8. We can format that as “pretty JSON” using a website like https://jsonformatter.org/.

9. Here is the generated command modified with JSON formatter.

Policy template

10. The above logs will confirm that our policy is working as expected.

Conclusion

The article covered some of the various scenarios a developer/user can face while writing KubeArmor policies and steps to troubleshoot the same. Our experience shows that the troubleshooting issues commonly fall under two categories:

  1. Incorrect format of policy and/or misspelled keywords.
  2. Missing out exact and/or dependent paths of binaries.

This article covered both the scenarios above in-depth and highlighted the steps needed to troubleshoot them respectively. An effective policy written without errors and misconfigurations will allow KubeArmor to protect your workloads!

To know more, connect with us using the social links given below.

KubeArmor website: https://kubearmor.com/

KubeArmor GitHub: https://github.com/kubearmor/KubeArmor

KubeArmor Slack: https://kubearmor.herokuapp.com/

Now you can protect your workloads in minutes using AccuKnox, it is available to protect your Kubernetes and other cloud workloads using Kernel Native Primitives such as AppArmor, SELinux, and eBPF.

Let us know if you are seeking additional guidance in planning your cloud security program.

Read more blogs from Cloud Security Category here.