GenAI System Layers

What the provider handles vs. what you build. Click any layer to expand. Filter by phase or use case.

Typically at PoC Typically at Pilot Typically at Production

Filter by use case

KYC — retail

KYC — insurer

AML triage

Reg. Q&A

⚡ Data ingestion

Document loader / parser

Extracts text from PDFs, Word, Excel, HTML. Where data quality problems first appear — garbage in, garbage out.

OCR

For scanned documents. Azure Document Intelligence, AWS Textract. Quality varies significantly by document age and scan quality.

Chunking strategy

How you split long documents before embedding. Fixed-size, semantic, or by section. Affects retrieval quality more than most people expect.

Data pipeline / ETL

Automated ingestion from source systems — DMS, SharePoint, email. Keeps content current without manual re-uploads.

→

⚡ Retrieval (RAG)

Dense retrieval

Vector similarity search over embeddings. Good for conceptual / semantic queries.

Sparse retrieval (BM25)

Traditional keyword search. Better than vector for exact terms — names, IBANs, regulation article numbers.

Hybrid search + re-ranker

Combines both retrieval methods, then re-scores with a cross-encoder. Meaningfully better recall at production scale.

Post-retrieval processing

Context compression, relevance filtering, citation tracking. Ensures chunks are actually useful before passing to the model.

→

⚡ Prompt management

Prompt versioning

Prompts stored in git or a config file. Prevents "who changed the prompt and why did it break?" Treat prompts as code.

Prompt templating

Variable injection, conditional blocks, dynamic few-shot selection, locale variants. Separates prompt logic from application logic.

Prompt testing

Regression suites on every prompt change. As important as unit tests for code.

↑

⚡ Memory

In-context memory

The live conversation window. Managed via truncation or summarisation of older turns.

Semantic memory (vector store)

Embeddings of documents, past conversations — queried by similarity. Pinecone, Weaviate, pgvector.

Episodic memory

Per-user history across sessions. Compressed into summaries so the model remembers without replaying full transcripts.

Structured memory

Explicit user profile: preferences, past decisions. Updated by extraction pipelines that pull structured facts out of conversations.

→

⚡ Foundation model (via API)

Pre-training & architecture

Transformer design, training data curation, distributed GPU training. Months of compute per model.

Alignment & safety tuning

RLHF, Constitutional AI, preference modelling, red-teaming. Ensures the model follows instructions without being harmful.

Serving infrastructure

Continuous batching, KV-cache, quantisation, tensor parallelism. What makes the API respond in milliseconds under load.

Embedding models

Convert text to vectors for semantic search. Provided as an API endpoint — no need to run your own.

Multimodal capabilities

Vision, audio transcription, code execution. Added at the model level — available via API without extra work.

Model versioning & deprecation

Versioned endpoints so your application doesn't break when a new model ships. You opt into upgrades.

→

⚡ Evaluation

Golden test set

Curated inputs with expected outputs. Run on every model or prompt change. Without this, you can't detect regressions.

LLM-as-judge

Use a second model to score outputs on correctness and groundedness. Scalable alternative to human evaluation.

Online evaluation

Implicit signals from production: thumbs up/down, copy actions, follow-up questions. Continuous feedback loop.

Human eval pipeline

Structured process for domain experts to rate sampled outputs. Required for high-stakes use cases.

🏗️ Observability

Distributed tracing

Full trace per request: input → retrieval → model call → tool calls → output. Essential for debugging agentic systems.

Cost tracking

Token cost per query, per user, per use case. Often the first surprise after go-live.

Accuracy monitoring

Golden test set run on a schedule. Catches silent model degradation when the provider updates the underlying model.

Canary deployment

Routes a small % of traffic to a new model version before full rollout. Standard IT playbook.

⚡ Orchestration

Workflow / chain orchestration

Multi-step pipelines: extract → classify → generate. LangChain, LlamaIndex, or custom.

Agent loop / tool use

Model calls external tools and iterates on results. Powerful but unpredictable — needs tight guardrails in regulated use cases.

Durable workflow engine

For long-running tasks: survives crashes, supports pause/resume, maintains audit trail. Temporal or similar.

🔒 Application & access

Auth + SSO

Integrate with existing identity (Azure AD, Okta). The moment you have more than one user.

API gateway / rate limiting

Controls access and prevents runaway cost from a bug or a bad actor. Token budgets per user and per tenant.

Integration connectors

Reads/writes to Salesforce, SAP, ERP, compliance portals. Each connector is a scoped project — often the largest cost item.

Caching layer

Semantic cache for similar queries, prefix cache for common system prompts. Cuts cost and latency at high volume.

🔒 Safety & compliance

Audit logging

Immutable log: user, timestamp, input, output, model version, latency. Non-negotiable in regulated industries past pilot.

PII detection / redaction

Strips personal data before it reaches a cloud API. Required if data is GDPR-regulated and the model is not on a private tenant.

Guardrails / output validation

Schema validation, confidence thresholds, hallucination checks. Cost scales with how low your tolerance is.

Prompt injection defense

Prevents adversarial inputs from hijacking model behavior. Scales in importance with user volume.

🏗️ Platform & DevOps — standard ICT, nothing GenAI-specific

CI/CD pipeline

GitHub Actions / Azure DevOps. Prompt changes are code changes — they go through the same pipeline.

Containerisation (Docker)

Reproducible environments. Required before any shared deployment.

Container orchestration (Kubernetes)

AKS / EKS. Auto-scaling, health checks, rolling deployments. Needed when load varies or uptime SLA > 99%.

Infrastructure as Code

Terraform / Bicep. No manual console changes in production. Required for audit trails in regulated environments.

Secret management

Azure Key Vault / HashiCorp Vault. API keys never in source code. Rotate without redeployment.

WAF + network security

Web Application Firewall, private endpoints, network segmentation. Mandatory for any external-facing deployment.

Vulnerability scanning

SAST, DAST, dependency audit (Snyk, Dependabot). Run automatically on every build. Required by DORA.

Penetration testing

Annual or per major release. Required by CSSF DORA. Scope includes prompt injection and model-specific attack surfaces.

Disaster recovery

RTO / RPO, backup strategy, tested failover. Business-critical internal tools: RTO < 4h, RPO < 1h.

Change management (ITIL)

CAB approval, rollback plan before deployment, post-implementation review. Applies to AI systems like any other.

Multi-region / failover

Required when uptime SLA ≥ 99.9%. Luxembourg data residency may limit region choices within EU.

Encryption at rest & in transit

TLS 1.2+ for all API calls, AES-256 for stored data. Table stakes for any regulated data.