Generative AI Interview Questions

1. When would you choose RAG over fine-tuning?

Use RAG when the knowledge changes often, must be cited, or is too large to bake into weights — like company documents or live data. Fine-tune when you need to change the model’s style, format, or behavior consistently, or teach a narrow skill. Many production systems combine both.

2. How does a RAG pipeline work end to end?

Documents are chunked, embedded, and stored in a vector database. At query time, the user’s question is embedded, the most similar chunks are retrieved, and they are inserted into the prompt as context. The LLM then generates an answer grounded in those chunks.

3. What is chunking and why does chunk size matter?

Chunking splits documents into smaller pieces before embedding. Too large, and retrieval becomes imprecise and wastes context; too small, and chunks lose meaning. Good chunking balances semantic completeness with retrieval precision, often with overlap between chunks.

4. What is LoRA and why is it used?

LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method that trains a small number of additional weights instead of the whole model. It dramatically reduces compute and memory cost while retaining most of the benefit of full fine-tuning.

5. What is PEFT?

PEFT (Parameter-Efficient Fine-Tuning) is a family of techniques — including LoRA and adapters — that fine-tune models by updating only a small subset of parameters, making customization cheaper and faster.

6. How do you reduce hallucinations in production?

Ground the model with RAG, instruct it to say “I don’t know” when unsure, lower temperature, add citations and verification steps, use guardrails to validate outputs, and keep the retrieved context tight and relevant.

7. What is RLHF?

Reinforcement Learning from Human Feedback aligns a model with human preferences. Humans rank model outputs, a reward model learns those preferences, and the LLM is optimized to produce more preferred responses.

8. What is the difference between dense and sparse retrieval?

Dense retrieval uses embeddings and semantic similarity. Sparse retrieval (like BM25) uses keyword matching. Hybrid search combines both to capture exact terms and semantic meaning.

9. How do you control the cost of an LLM application?

Reduce token usage with concise prompts and retrieval, cache frequent responses, use smaller models for simple tasks, batch requests, and route easy queries to cheaper models while reserving large models for hard ones.

10. What is a reranker in RAG?

A reranker is a second-stage model that reorders retrieved chunks by relevance to the query. It improves answer quality by ensuring the most useful context appears first, since initial vector retrieval is not always precise.

11. What is the ReAct pattern in AI agents?

ReAct interleaves reasoning and acting: the agent thinks about what to do, takes an action (like a tool call), observes the result, and repeats until it reaches the goal. It makes agent behavior more reliable and explainable.

12. How do you evaluate a RAG system? 

Measure retrieval quality (are the right chunks fetched?) and generation quality (is the answer faithful to the context and relevant to the question?). Metrics include context precision/recall, faithfulness, and answer relevance, often using LLM-based evaluation.

13. What is quantization?

Quantization reduces the numerical precision of model weights (for example, from 16-bit to 4-bit) to shrink size and speed up inference, usually with a small accuracy trade-off. It enables running large models on limited hardware.

14. What is a context window limitation, and how do you handle long documents?

Models can only process a fixed number of tokens at once. For longer content, you chunk and retrieve relevant parts (RAG), summarize hierarchically, or use models with larger context windows.

15. What are guardrails?

Guardrails are rules and checks that constrain model inputs and outputs — filtering unsafe content, validating formats, enforcing topic boundaries, and preventing prompt injection.

16. What is prompt injection and how do you defend against it?

Prompt injection is when malicious input tries to override the system’s instructions. Defenses include separating trusted instructions from untrusted data, input/output validation, least-privilege tool access, and not blindly trusting retrieved or user-supplied text.

17. What is the difference between embeddings models and generation models?

Embedding models convert text into vectors for search and comparison. Generation models produce text. RAG uses an embedding model for retrieval and a generation model to write the final answer.

18. How do agentic workflows differ from a single LLM call?

A single call returns one response. An agentic workflow loops — planning, calling tools, observing results, and refining — to handle multi-step tasks. It is more capable but harder to control, debug, and cost-manage.

19. What is model distillation?

Distillation trains a smaller “student” model to mimic a larger “teacher” model, producing a faster, cheaper model that retains much of the larger model’s capability.

20. How do you keep an LLM application’s knowledge up to date?

Use RAG with a regularly updated knowledge base rather than retraining. Refresh embeddings when documents change, version your data, and monitor for stale or conflicting information.