Join Vultr Community
Points, levels & badges
Enter challenges & submit builds
Register to events & meetups
Get your work featured
Hi everyone!
I'm currently building Kubeagent, an autonomous Linux incident response AI agent designed to monitor Kubernetes workloads, analyze logs, and execute automated remediation workflows (like restarting pods, scaling deployments, or patching configurations).
While testing the agent's multi-step debugging loops, I ran into a core security and architecture challenge: How do you safely isolate and sandbox LLM-generated kubectl commands or remediation scripts before they hit a production cluster?
Currently, I'm exploring a few approaches:
My Questions:
Points, levels & badges
Enter challenges & submit builds
Register to events & meetups
Get your work featured
Hey Maaz,
This is an interesting problem, especially when you're building an autonomous incident response agent. I think the biggest security risk here is not necessarily the LLM generating an incorrect command. It is allowing that command to execute with permissions that are broader than what the actual remediation requires.
I would approach this by separating the agent's decision-making process from the execution layer. The agent should be able to investigate an incident, understand the cluster state, and propose a remediation, but the component that actually interacts with Kubernetes should enforce strict permissions and validation independently of the LLM.
Here is how I would design it.
This would be my first recommendation.
Rather than allowing the agent to generate arbitrary shell commands and passing them directly to a terminal, I would introduce a predefined set of remediation actions.
For example, instead of allowing the LLM to generate:
I would expose a structured tool interface such as:
The execution service would validate the action, check whether the requested resource is within the agent's permitted scope, and then execute the corresponding Kubernetes API operation.
I would maintain an explicit allowlist of supported actions, such as:
Anything outside this list should be rejected by default.
This also makes it much easier to audit what the agent is actually doing.
I completely agree with your approach of using restricted ServiceAccounts. However, I would take it a step further and avoid giving the main AI agent any direct write access to the production cluster.
I would separate the architecture into two components:
Agent service: Responsible for analyzing logs, identifying problems, and proposing remediation actions. Its Kubernetes identity should ideally be read-only.
Remediation controller: A separate service responsible for executing approved actions. It receives structured requests and has narrowly scoped permissions.
For example, I would create separate ServiceAccounts for different environments or remediation responsibilities rather than giving one account access to the entire cluster.
Use namespace-scoped Roles and RoleBindings wherever possible instead of broad ClusterRoleBindings.
I would also avoid wildcard permissions such as:
In particular, the remediation identity should not have permissions to:
One subtle point worth mentioning is that Kubernetes RBAC operates at the resource and verb level. It cannot, by itself, enforce every business rule around which fields an application is permitted to modify. That is why I would combine RBAC with application-level validation and admission policies.
Documentation:
https://kubernetes.io/docs/concepts/security/rbac-good-practices/
Your idea of using gVisor is worth exploring.
A normal container shares the host kernel, so simply placing an LLM-generated script inside a Docker container does not provide the same isolation as running it inside a dedicated sandbox.
gVisor takes a different approach by implementing a Linux-like application kernel in userspace. Its runsc runtime integrates with container tooling, including Kubernetes.
For situations where you genuinely need to execute dynamically generated scripts, I would consider the following configuration:
Most importantly, I would not mount the production cluster's ServiceAccount token inside this sandbox.
The sandbox should execute and validate the script without possessing the credentials required to modify production resources. The actual remediation should go through the separate controller.
gVisor documentation:
https://gvisor.dev/docs/
Kubernetes Pod Security Standards:
https://kubernetes.io/docs/concepts/security/pod-security-standards/
This is another layer I would strongly recommend.
Before a remediation reaches the Kubernetes API server, it should pass through a policy-validation mechanism.
For this, I would look at:
For example, you could establish policies that prevent automated remediation from:
Kubernetes ValidatingAdmissionPolicy is particularly interesting because it provides an in-process way to enforce declarative policies using CEL.
For example, a policy can reject a deployment update when its requested replica count exceeds an approved maximum.
This is useful because even if the agent or remediation controller makes a mistake, the admission layer provides another opportunity to reject an unsafe change.
Documentation:
https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/
I would not necessarily put every remediation behind a human approval gate.
During an actual production incident, requiring approval for every small operation can significantly increase MTTR.
Instead, I would classify remediation actions based on their potential impact.
Low-risk actions:
Examples include collecting logs, inspecting events, checking pod health, and restarting a specific non-critical workload within predefined limits.
These could be automatically executed if they pass the policy checks.
Medium-risk actions:
Examples include scaling production deployments, restarting critical services, or modifying deployment configurations.
These could require additional validation, health checks, and approval depending on the environment.
High-risk actions:
Examples include deleting persistent volumes, changing RBAC permissions, modifying network policies, deleting namespaces, or modifying cluster-wide configurations.
I would require explicit human approval for these operations.
There is one additional detail I would implement: approval should be tied to the exact proposed action, target resource, and configuration diff, rather than simply approving an entire agent execution session.
It should also expire after a short period. Before execution, the controller should revalidate the resource state to ensure that the approved change has not become stale.
That prevents an agent from obtaining approval for one operation and subsequently executing a different operation using the same approval.
Before making actual changes, I would run the proposed Kubernetes operation through server-side dry-run validation wherever supported.
For example:
This sends the request through server-side validation and admission processing without persisting the resource.
However, it is important to understand that a successful dry run does not guarantee that the remediation is operationally safe. It validates the API request, not the real-world effect of restarting a service or changing its replica count.
I would combine this with a few additional checks:
I would also maintain a complete audit trail for every action.
For each remediation, I would record the incident ID, agent's reasoning or summarized justification, proposed action, affected resources, policy-validation results, approval identity, execution timestamp, and final outcome.
Sensitive information from logs and configuration should be redacted.
For reversible deployment changes, I would preserve the previous configuration and associate it with the remediation ID so the controller can attempt a controlled rollback if post-remediation health checks fail.
I would also avoid assuming that every action is automatically reversible. Deleting a persistent volume or changing certain security policies can have consequences that a rollback cannot undo.
I think the best approach is to make safety checks deterministic and relatively inexpensive, rather than introducing a second LLM-based approval process.
The execution flow could look something like this:
I would also introduce an idempotency mechanism so that if the agent retries the same remediation because of a timeout, it does not unintentionally execute the operation multiple times.
For example, two concurrent remediation loops should not repeatedly restart the same deployment.
A lock or lease associated with the affected resource and incident ID can help prevent conflicting operations.
If I were implementing Kubeagent, I would probably go with the following combination:
LayerTechnology / ApproachAgent reasoningLLM with structured tool callingExecution interfaceDedicated remediation controllerAccess controlKubernetes RBACScript isolationgVisorPolicy enforcementKyverno or OPA GatekeeperNative policy validationValidatingAdmissionPolicy (CEL)Network isolationKubernetes NetworkPoliciesHuman approvalWebhook-based approval serviceMonitoringPrometheus and GrafanaAudit trailKubernetes audit logs and centralized application logs
I would also keep the remediation controller outside the agent's execution environment and ensure it has its own tightly controlled identity.
I think the interesting design challenge here is that an autonomous incident response system should not be trusted simply because its reasoning is good.
Even a well-performing model can misunderstand an incident, misinterpret logs, or propose a remediation that fixes one problem while introducing another.
I would therefore make the execution layer completely deterministic, independently secured, and policy-driven. Let the LLM decide what it thinks should be done, but let the controller decide whether that action is actually permitted.
That way, you can still automate common incident-response operations and reduce MTTR without giving an AI agent unrestricted control over your production infrastructure.
I'd be particularly interested in how you're planning to handle the boundary between the agent and the remediation controller. I think that separation is ultimately more important than choosing between Docker and gVisor alone.
To safely isolate and execute AI-driven Kubernetes remediation:
runsc) containers or Firecracker MicroVMs to isolate the host kernel.