Autonomous Cloud Compliance & AI Auditing Engine

Problem

Enterprise customers send long security questionnaires, such as CAIQ, SOC 2, and SIG reviews, often with 250 or more questions. Answering them by hand means checking live cloud configuration, searching policy documents, and writing up each answer with evidence. It is slow, and a generic AI answer is worse than none, because a confident answer with no real evidence behind it is a liability.

Context

A self-built project that went through two versions. The first version automated AWS infrastructure auditing and reporting. The second turned it into an evidence-based engine that answers security questionnaires from live AWS evidence and policy manuals. It supports security reviews; it is not a formal compliance certification or an independent attestation.

My Role

I designed and built both versions end-to-end.

Architecture

First version: automated AWS audit

1. A Python and Flask service in an isolated Docker container runs read-only AWS CLI audit commands and normalizes the JSON output.
2. n8n orchestrates extraction, evaluation, and storage.
3. A context-injected Langflow agent, using Groq, evaluates the collected configuration against security checkpoints.
4. Findings, evidence, and remediation roadmaps are stored in Supabase.

Second version: evidence-based questionnaire engine

1. Evidence collection. 50+ live AWS CIS Benchmark checks, run through the AWS CLI sidecar.
2. Document retrieval. Policy manuals parsed with PyMuPDF, embedded locally with FastEmbed (all-MiniLM-L6-v2), and stored in Supabase with pgvector.
3. Answering. A 6-node LangGraph state machine using Corrective RAG (CRAG). Retrieved passages are scored for relevance by Llama 3.1 8B on Groq, and weak retrievals are corrected before an answer is written.
4. Verification. A Critic Agent checks that every answer cites a real AWS ARN or a real document section before it's accepted.
5. Queueing. An asynchronous worker in n8n and PostgreSQL feeds questions through the engine, served by FastAPI.

Technology Stack

LangGraph, LangChain, CRAG, LlamaIndex, Groq (Llama 3.1 8B), Supabase pgvector, FastEmbed, PyMuPDF, FastAPI, n8n, PostgreSQL, Docker, AWS CLI. The first version also used Langflow and Flask.

Implementation

• Every answer is required to carry a citation: a verified AWS ARN from the live account or a specific section of a policy document
• Relevance scoring decides whether retrieved context is good enough, or whether retrieval needs to be corrected first
• Embeddings run locally, so there's no paid embedding API
• The queue worker uses row-level processing locks so two workers never take the same question, 3-second delays to respect rate limits, and exponential backoff on failures

Reliability / QA

The design separates "collect" from "judge" and "judge" from "accept". The model only sees evidence that was actually collected or retrieved. The Critic Agent rejects answers whose citations don't check out. Queue locks and backoff keep long runs stable instead of failing on rate limits.

Challenges

• Making AI answers traceable to real evidence rather than plausible text
• Retrieving the right policy passage out of long PDF manuals
• Running thousands of questions without hitting model rate limits or timeouts

Solution

Evidence first, then retrieval with correction, then a critic that verifies citations, all driven by a queue that respects rate limits.

Results

• Automates answering of 250+ question enterprise security reviews (CAIQ, SOC 2, SIG)
• ~90% reduction in questionnaire turnaround time, as reported
• 0.95 semantic relevance score for retrieval, as reported, with no embedding API cost
• 2,000+ questions processed with no timeout failures, as reported
• The first version produces a 100-point infrastructure assessment with a 0–100 risk score and a remediation roadmap

A recorded walkthrough of the first version is linked below. The second version doesn't have a public demo yet.

Lessons Learned

For compliance work, an AI answer is only useful if someone can check it. Requiring a verifiable citation for every answer, and rejecting the ones that fail, turned the model from a writer into a researcher whose work can be reviewed.