Inference Infrastructure & Serving Security

Best time to learn: During the existing production-engineering and deployment weeks, after Docker, networking, authentication, and observability. This is a prerequisite for operating self-hosted or multi-tenant open-weight models.

Prerequisites - Docker and container basics. - HTTP/API authentication and authorization. - Network segmentation, secrets management, and logging. - Basic inference-server architecture.

### Threat model

The model-serving layer can be attacked even if the model's prompt handling is excellent. A reachable unauthenticated endpoint, weak container, overprivileged process, or shared-resource leak can compromise the system.

Study: - Inference gateway versus model server responsibilities. - vLLM or another serving runtime at an architectural level. - Authentication, authorization, per-tenant routing, and request identity propagation. - Network segmentation and outbound egress restrictions. - Container hardening, non-root execution, read-only filesystems, and capability reduction. - Model file mounts and runtime filesystem access. - Secret injection and rotation. - GPU/resource isolation and multi-tenant considerations. - Request concurrency, queue limits, memory pressure, and graceful shutdown. - Endpoint exposure, health endpoints, metrics endpoints, and administrative interfaces. - Cloud IAM and workload identity where applicable.

### Labs

• Serving topology (2h): Diagram gateway, model server, storage, telemetry, network boundaries, and identity propagation. Identify every externally reachable endpoint.

• Endpoint hardening (2h): Put a local inference endpoint behind an authenticated gateway. Verify unauthorized and cross-tenant requests are rejected.

• Container hardening (2h): Run as non-root, remove unnecessary capabilities, restrict mounts, avoid privileged mode, and use a read-only filesystem where feasible.

• Network and secrets (2h): Restrict egress to required destinations; verify secrets are not baked into images, logs, or environment dumps.

• Resource isolation (2h): Add request size, concurrency, queue, and timeout limits. Simulate resource pressure and verify controlled rejection and recovery.

### Project integration

• Project 1: If Ollama or another local runtime is exposed beyond localhost, put it behind an explicit access boundary. Never assume a local inference port is safe to expose.

• Project 2: Harden the deployed API, embedding/reranking workers, vector store, and model endpoint as separate components.

• Project 3: Keep agent execution workers isolated from the model-serving process and from application secrets.

### Measurable completion criteria

• No model-serving or administrative endpoint is unintentionally public.

• Unauthorized requests and cross-tenant access fail closed.

• Containers run with documented minimum privileges.

• Egress and secret-handling policies are tested.

• Resource-exhaustion tests produce bounded failure rather than uncontrolled service degradation.

Do not overreach: Do not deploy Kubernetes, a GPU cluster, or a service mesh solely for résumé value. A hardened single-host deployment can demonstrate the relevant controls.

------------------------------------------------------------------------