How to Troubleshoot KubeArmor policies?

In this blog, we are going to see how to troubleshoot KubeArmor policies in
GKE.

This blog assumes that you have read the concepts which explain all the
components and fundamentals of KubeArmor. The blog will cover the following
key steps:

• Creating policies
• Applying policies
• Testing policies
• Troubleshooting policies

Creating policies

1. To deploy KubeArmor in GKE apply this script. https://raw.githubusercontent.com/kubearmor/KubeArmor/master/deployments/GKE/kubearmor.yaml
2. Another method is to deploy kubearmor in GKE follow this guide to deploy kubearmor in a very simple way.
3. Sample PostgreSQL policy is given below:

   
   Sample PostgreSQL policy

4. The above policy has been created for the PostgreSQL workload. Psql is the
   binary for PostgreSQL. If anyone tries to access the psql the policy will
   deny the access. Those binaries will be found easily in the bin folder
   inside the Linux directory.
5. Let’s apply the policy and see it in action!

Applying policies

1. When applying policies make sure to verify with
   YAML lint applications to verify the structure and indentation of the YAML files. If
   not verified, the policy will fail to apply.
   Note:
   The policy must follow the required policy structure of KubeArmor.
   View here
   

   KubeArmor structure

2. The above error shows that you may have missed the YAML indentation. As
   shown below.
   
3. The above error can simply be fixed by changing the indentation. Make sure
   to use appropriate YAML lint extensions in your code editor to avoid these
   kinds of errors.
   
4. Now we have fixed the policy. Let’s try to apply that policy and see it in
   action.

   [root@karmor]$ kubectl apply -f postgresql-policy.yaml
   kubearmorpolicy.security.kubearmor.com/ksp-stigs-postgresql-console
   created</span >

5. Now our policy has been created and applied in the GKE.
6. Check here for a complete guide and specification to write a kubearmor policy.

Testing policies

1. When you are testing policies you have to check with KubeArmor Security
   policy specifications.
   https://github.com/kubearmor/KubeArmor/blob/main/getting-started/security_policy_specification.md
2. Make sure the wordings are spelled correctly. Otherwise, it throws an error
   like this shown below. These types of errors are likely to happen when we
   misspell words. So make sure to check in with the contribution guide
   mentioned above.
   

   Testing policy

3. The above error shows we misspelled the words “matchPath” instead of
   “matchPaths” and followed by “matchLabel” instead of “matchLabels” also
   “tag” instead of “tags”. Let’s fix those errors and test the policy again
4. This is the fixed policy with correct spellings.
   

   Kubearmor policy

   

   Code block

5. Now our policy is applied and ready.

   [root@kubearmor]$ kubectl apply -f postgres-policy.yaml
   kubearmorpolicy.security.kubearmor.com/ksp-stigs-postgresql
   created</span >

6. Let’s deploy PostgreSQL deployment in our cluster to test the policy.
7. To deploy a Postgres service in your cluster we have to apply this script
   

   Script-1

   

   Script-2

   

   Script-3

8. Now we have our working Postgres pod running in our k8s cluster.
   

   Postgres pod

9. Now let’s try to access the psql binary to test if it’s working or not. The
   goal is to block access to the psql binary.
   

   psql binary

10. As you can see in this above problem, psql binary is still accessible even
    after applying the policy. That policy is not working as we expected. Let’s
    troubleshoot where it went wrong and fix the policy.

Troubleshooting Policies

1. Even after using the correct format and parameters, the policy is not
   working and psql binary is still accessible. Why?
2. Let’s check the binary location with commands we know such as “whereis” and
   “which”. So we can figure out whether we added the correct path or not.
3. Here we checked psql binary location “whereis” command It shows that we
   added the correct path in our policy but still the policy is not working as
   we expected.
   
4. Let’s dive in deep and find if we have any other connections to the psql
   binary. If there is, we can add that to our policy.
   

   Kubearmor pod

5. Now when we check the psql with “ls -la” command. That psql binary is linked
   with “../share/postgresql-common/pg_wrapper”
6. So now we can see that it’s linked with another binary. So that’s why it’s
   not blocking the psql binary. We can add that in our policy and test that
   again.
7. Here’s the updated policy with that path included
   

   Kubearmor policy

8. Let’s apply this policy and check whether it is blocking the psql binary or
   not.
9. Now our policy is created without any errors.
   
   

   Policy template

10. Now we have our working policy that can block the psql binary in postgres
    pod. So we can authorize the use of the postgres server.

KubeArmor Logs

1. To check kubearmor generated logs. We have to type this command to see that.
2. kubectl -n kube-system get pods -A | grep kubearmor
3. This command will list all kubearmor pods as shown below.
   

   Kubearmor pods

4. You have to select the first three pods that are named “kubearmor-1234”
   similarly. This name will differ in your environment. You have to enter the
   command shown below.
5. kubectl -n kube-system exec -it <kube-armor-pod> — tail
   /tmp/kubearmor.log
6. You will see the logs like this in the terminal as shown in the image below

   

   Kubearmor pod

7. We can format that as “pretty JSON” using a website like
   https://jsonformatter.org/.
8. Here is the generated command modified with JSON formatter.
   

   Policy template

9. The above logs will confirm that our policy is working as expected.

   

   psql binary

10. As you can see in this above problem, psql binary is still accessible even
    after applying the policy. That policy is not working as we expected. Let’s
    troubleshoot where it went wrong and fix the policy.