# NGINX OpenTelemetry 可观测性指南 本文档介绍如何在 NGINX 中使用 OpenTelemetry 模块实现分布式追踪和可观测性。 ## 目录 1. [OpenTelemetry 概述](#opentelemetry-概述) 2. [模块指令参考](#模块指令参考) 3. [分布式追踪配置](#分布式追踪配置) 4. [与 Jaeger/Zipkin 集成](#与-jaegerzipkin-集成) 5. [自定义属性和事件](#自定义属性和事件) 6. [完整配置示例](#完整配置示例) 7. [最佳实践](#最佳实践) --- ## OpenTelemetry 概述 ### 什么是 OpenTelemetry OpenTelemetry 是一个开源的可观测性框架，提供标准化的 API、库和工具来收集分布式追踪、指标和日志数据。它由 Cloud Native Computing Foundation (CNCF) 托管，是 Prometheus、Jaeger 和 OpenCensus 等项目合并后的统一解决方案。 ### 核心概念 | 概念 | 描述 | |------|------| | **Trace** | 分布式追踪，表示请求在系统中的完整调用链路 | | **Span** | 追踪中的基本工作单元，包含操作名称、起止时间、属性等 | | **Context** | 追踪上下文，用于在服务间传播追踪信息（traceparent/tracestate） | | **Resource** | 描述产生遥测数据的实体（如服务名称、版本、主机） | | **Exporter** | 将遥测数据发送到后端存储（如 OTLP、gRPC） | ### 架构流程 ``` ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ Client │───▶│ NGINX │───▶│ Backend │───▶│Database │ └─────────┘ └────┬────┘ └─────────┘ └─────────┘ │ ▼ ┌────────────────┐ │ ngx_otel_module │ └───────┬────────┘ │ ▼ ┌───────────────┐ ┌───────────┐ ┌──────────┐ │OTEL Collector │───▶│ Jaeger │ │ Zipkin │ └───────────────┘ └───────────┘ └──────────┘ ``` ### 模块版本要求 - NGINX Plus R28 或更高版本 - `ngx_otel_module` 动态模块（从源码编译或 NGINX Plus 包含） --- ## 模块指令参考 ### otel_exporter 配置 OpenTelemetry 数据导出参数。 | 指令 | 语法 | 默认值 | 上下文 | |------|------|--------|--------| | `otel_exporter` | `{ ... }` | — | `http` | **子指令：** | 子指令 | 语法 | 默认值 | 描述 | |--------|------|--------|------| | `endpoint` | `[(http\|https)://]host:port;` | — | OTLP/gRPC 端点地址 | | `trusted_certificate` | `path;` | 系统 CA | PEM 格式 CA 证书文件（v0.1.2+） | | `header` | `name value;` | — | 自定义 HTTP 请求头 | | `interval` | `time;` | `5s` | 导出最大间隔时间 | | `batch_size` | `number;` | `512` | 每批次最大 Span 数量 | | `batch_count` | `number;` | `4` | 每个 worker 的待处理批次数 | **示例：** ```nginx http { otel_exporter { endpoint otel-collector:4317; interval 5s; batch_size 512; batch_count 4; trusted_certificate /etc/nginx/certs/ca.pem; header X-API-Key secret_key; } } ``` ### otel_service_name 设置 OTel Resource 的 `service.name` 属性。 | 指令 | 语法 | 默认值 | 上下文 | |------|------|--------|--------| | `otel_service_name` | `name;` | `unknown_service:nginx` | `http` | **示例：** ```nginx http { otel_service_name nginx-gateway; } ``` ### otel_resource_attr 设置自定义 OTel Resource 属性（v0.1.2+）。 | 指令 | 语法 | 默认值 | 上下文 | |------|------|--------|--------| | `otel_resource_attr` | `name value;` | — | `http` | **示例：** ```nginx http { otel_resource_attr deployment.environment production; otel_resource_attr service.version 1.2.3; otel_resource_attr host.name $hostname; } ``` ### otel_trace 启用或禁用 OpenTelemetry 追踪。 | 指令 | 语法 | 默认值 | 上下文 | |------|------|--------|--------| | `otel_trace` | `on \| off \| $variable;` | `off` | `http`, `server`, `location` | **示例：** ```nginx http { otel_trace off; server { listen 80; otel_trace on; location /api { otel_trace on; } location /health { otel_trace off; # 健康检查不记录 } } } ``` ### otel_trace_context 配置 traceparent/tracestate 头的传播方式。 | 指令 | 语法 | 默认值 | 上下文 | |------|------|--------|--------| | `otel_trace_context` | `extract \| inject \| propagate \| ignore;` | `ignore` | `http`, `server`, `location` | **选项说明：** | 值 | 描述 | |----|------| | `extract` | 从入站请求中提取追踪上下文，继承上游标识符 | | `inject` | 向出站请求注入新的追踪上下文，覆盖现有上下文 | | `propagate` | 更新现有上下文（先 extract 再 inject），保持追踪链完整 | | `ignore` | 忽略上下文头处理 | **示例：** ```nginx server { location / { # 作为入口网关，注入新追踪上下文 otel_trace_context inject; proxy_pass http://backend; } location /api/ { # 作为中间代理，传播上游追踪上下文 otel_trace_context propagate; proxy_pass http://api_backend; } } ``` ### otel_span_name 定义 OTel Span 的名称。 | 指令 | 语法 | 默认值 | 上下文 | |------|------|--------|--------| | `otel_span_name` | `name;` | location 名称 | `http`, `server`, `location` | **示例：** ```nginx server { location /api/users { otel_span_name "GET /api/users"; # 或使用变量 otel_span_name "$request_method $uri"; } } ``` ### otel_span_attr 添加自定义 OTel Span 属性。 | 指令 | 语法 | 默认值 | 上下文 | |------|------|--------|--------| | `otel_span_attr` | `name value;` | — | `http`, `server`, `location` | **示例：** ```nginx server { location /api/ { otel_span_attr http.route "/api/*"; otel_span_attr user.id $remote_user; otel_span_attr client.ip $remote_addr; } } ``` ### 嵌入式变量 | 变量 | 描述 | |------|------| | `$otel_trace_id` | 追踪标识符 | | `$otel_span_id` | 当前 Span 标识符 | | `$otel_parent_id` | 父 Span 标识符 | | `$otel_parent_sampled` | 父 Span 的采样标志（`1` 或 `0`） | --- ## 分布式追踪配置 ### Trace 上下文传播 追踪上下文传播是分布式追踪的核心，确保请求在多个服务间保持相同的追踪标识。 #### W3C Trace Context 标准 NGINX 使用 W3C Trace Context 标准： - **traceparent**: `00-{trace-id}-{parent-id}-{flags}` - **tracestate**: 厂商特定的上下文信息 ``` Traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01 │ │ │ │ │ │ │ │ │ └── 标志位( sampled: 01) │ │ │ └── 父 Span ID │ │ └── Trace ID │ └── 版本 └── 固定前缀 ``` #### 传播模式配置 **场景 1: 边缘网关（追踪入口）** ```nginx http { otel_service_name nginx-edge-gateway; otel_trace on; server { listen 80; server_name api.example.com; location / { # 注入新的追踪上下文 otel_trace_context inject; # 将追踪 ID 传递给后端 proxy_set_header X-Trace-ID $otel_trace_id; proxy_set_header X-Span-ID $otel_span_id; proxy_pass http://backend_cluster; } } } ``` **场景 2: 中间代理（追踪传播）** ```nginx server { listen 8080; location / { # 传播上游追踪上下文 otel_trace_context propagate; # 将追踪头传递给下游服务 proxy_set_header traceparent $http_traceparent; proxy_set_header tracestate $http_tracestate; proxy_pass http://internal_services; } } ``` **场景 3: 混合模式** ```nginx server { location /public/ { # 公共 API: 创建新追踪 otel_trace_context inject; proxy_pass http://public_backend; } location /internal/ { # 内部服务: 传播已有追踪 otel_trace_context propagate; proxy_pass http://internal_backend; } location /health { # 健康检查: 忽略追踪 otel_trace off; return 200 "healthy\n"; } } ``` ### Span 配置 #### 标准 Span 属性 NGINX 自动记录的 Span 属性： | 属性 | 描述 | 示例值 | |------|------|--------| | `http.method` | HTTP 方法 | GET, POST, PUT | | `http.url` | 请求 URL | `https://api.example.com/users` | | `http.scheme` | 协议 | http, https | | `http.host` | 主机名 | `api.example.com` | | `http.status_code` | 响应状态码 | 200, 404, 500 | | `http.user_agent` | 用户代理 | Mozilla/5.0... | | `http.request_content_length` | 请求体大小 | 1024 | | `http.response_content_length` | 响应体大小 | 2048 | | `net.peer.ip` | 客户端 IP | 192.168.1.100 | | `net.peer.port` | 客户端端口 | 54321 | #### 自定义 Span 名称 ```nginx map $request_method $span_name { default "$request_method $uri"; GET "get_request"; POST "create_resource"; } server { location /api/ { otel_span_name $span_name; proxy_pass http://backend; } } ``` #### 条件性 Span 属性 ```nginx map $status $error_type { ~^[45] "client_or_server_error"; default ""; } server { location / { otel_span_attr error.class $error_type; otel_span_attr request.id $request_id; otel_span_attr tenant.id $http_x_tenant_id; proxy_pass http://backend; } } ``` ### 采样策略 采样控制追踪数据的收集量，平衡可观测性和性能开销。 #### 采样类型 | 采样类型 | 描述 | 使用场景 | |----------|------|----------| | **Head-Based** | 在追踪开始时决定采样 | 低延迟、低资源开销 | | **Tail-Based** | 基于完整追踪数据决定 | 捕获错误/慢请求 | | **Parent-Based** | 继承父 Span 的采样决定 | 保持追踪完整性 | #### 配置示例 **1. 始终采样（开发/测试环境）** ```nginx http { otel_trace on; # 所有请求都记录 } ``` **2. 比例采样（基于变量）** ```nginx # 使用 Lua 或外部模块实现比例采样 # 这里展示基于 Nginx 变量的实现 split_clients "$remote_addr$request_id" $trace_sampled { 10% "1"; # 10% 采样率 * "0"; # 90% 不采样 } server { location / { otel_trace $trace_sampled; proxy_pass http://backend; } } ``` **3. 基于请求特征采样** ```nginx map $uri $should_trace { default "0"; ~*\.html$ "1"; # 采样 HTML 页面 /api/critical/ "1"; # 采样关键 API /api/payment/ "1"; # 采样支付相关 } map $http_x_debug $force_trace { default ""; true "1"; } server { location / { # 优先使用 debug header，其次基于 URI otel_trace $force_trace$should_trace; proxy_pass http://backend; } } ``` **4. 错误/慢请求采样（结合 OpenTelemetry Collector）** ```yaml # otel-collector-config.yaml processors: tail_sampling: policies: - name: slow_requests type: latency latency: {threshold_ms: 500} - name: errors type: status_code status_code: {status_codes: [500, 502, 503, 504]} - name: probabilistic type: probabilistic probabilistic: {sampling_percentage: 10} ``` --- ## 与 Jaeger/Zipkin 集成 ### Jaeger 集成 #### 方法 1: Jaeger 原生 OTLP（推荐） Jaeger 1.35+ 原生支持 OTLP 协议。 **docker-compose.yaml:** ```yaml version: "3.8" services: jaeger: image: jaegertracing/all-in-one:1.60.0 container_name: jaeger ports: - "16686:16686" # Jaeger UI - "4317:4317" # OTLP gRPC - "4318:4318" # OTLP HTTP environment: - COLLECTOR_OTLP_ENABLED=true networks: - observability nginx: image: nginx:alpine container_name: nginx volumes: - ./nginx.conf:/etc/nginx/nginx.conf:ro ports: - "80:80" depends_on: - jaeger networks: - observability networks: observability: driver: bridge ``` **nginx.conf:** ```nginx load_module modules/ngx_otel_module.so; events { worker_connections 1024; } http { # OTLP 导出器配置 otel_exporter { endpoint jaeger:4317; interval 5s; batch_size 512; } # 服务标识 otel_service_name nginx-gateway; otel_resource_attr deployment.environment production; otel_resource_attr host.name $hostname; # 启用追踪 otel_trace on; server { listen 80; server_name localhost; location / { otel_trace_context inject; otel_span_name "$request_method $uri"; # 传递追踪上下文给后端 proxy_set_header traceparent $http_traceparent; proxy_set_header tracestate $http_tracestate; proxy_pass http://backend; } location /jaeger { # 返回当前追踪信息（调试用途） default_type application/json; return 200 '{"trace_id":"$otel_trace_id","span_id":"$otel_span_id"}'; } } } ``` #### 方法 2: 通过 OpenTelemetry Collector 用于需要额外处理的场景（过滤、转换、批处理）。 **docker-compose.yaml:** ```yaml version: "3.8" services: otel-collector: image: otel/opentelemetry-collector-contrib:0.117.0 container_name: otel-collector command: ["--config=/etc/otel-collector-config.yaml"] volumes: - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro ports: - "4317:4317" # OTLP gRPC - "4318:4318" # OTLP HTTP - "9464:9464" # Prometheus metrics networks: - observability jaeger: image: jaegertracing/all-in-one:1.60.0 container_name: jaeger ports: - "16686:16686" environment: - COLLECTOR_OTLP_ENABLED=true networks: - observability nginx: image: nginx:alpine container_name: nginx volumes: - ./nginx.conf:/etc/nginx/nginx.conf:ro ports: - "80:80" depends_on: - otel-collector networks: - observability networks: observability: driver: bridge ``` **otel-collector-config.yaml:** ```yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: batch: timeout: 1s send_batch_size: 1024 resource: attributes: - key: environment value: production action: upsert tail_sampling: policies: - name: slow_requests type: latency latency: {threshold_ms: 500} - name: errors type: status_code status_code: {status_codes: [500, 502, 503, 504]} exporters: otlp/jaeger: endpoint: jaeger:4317 tls: insecure: true debug: verbosity: detailed service: pipelines: traces: receivers: [otlp] processors: [batch, resource, tail_sampling] exporters: [otlp/jaeger, debug] ``` ### Zipkin 集成 #### 方法 1: 通过 OpenTelemetry Collector **docker-compose.yaml:** ```yaml version: "3.8" services: zipkin: image: openzipkin/zipkin:3 container_name: zipkin ports: - "9411:9411" environment: - STORAGE_TYPE=mem networks: - observability otel-collector: image: otel/opentelemetry-collector-contrib:0.117.0 container_name: otel-collector command: ["--config=/etc/otel-collector-config.yaml"] volumes: - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro ports: - "4317:4317" - "4318:4318" depends_on: - zipkin networks: - observability nginx: image: nginx:alpine container_name: nginx volumes: - ./nginx.conf:/etc/nginx/nginx.conf:ro ports: - "80:80" depends_on: - otel-collector networks: - observability networks: observability: driver: bridge ``` **otel-collector-config.yaml:** ```yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: batch: timeout: 1s send_batch_size: 1024 exporters: zipkin: endpoint: http://zipkin:9411/api/v2/spans format: json debug: verbosity: detailed service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [zipkin, debug] ``` #### 方法 2: Zipkin 直接接收 如果您的系统已使用 Zipkin，可以让 Collector 同时接收 OTLP 和 Zipkin 格式。 ```yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 zipkin: endpoint: 0.0.0.0:9411 processors: batch: exporters: zipkin: endpoint: http://zipkin:9411/api/v2/spans service: pipelines: traces: receivers: [otlp, zipkin] processors: [batch] exporters: [zipkin] ``` --- ## 自定义属性和事件 ### 自定义 Span 属性 #### 静态属性 ```nginx http { otel_resource_attr service.namespace ecommerce; otel_resource_attr service.version 2.1.0; server { location /api/ { otel_span_attr api.version v1; otel_span_attr team backend; } } } ``` #### 动态属性（使用变量） ```nginx map $request_time $latency_bucket { ~^0\.[0-4] "fast"; ~^0\.[5-9] "medium"; default "slow"; } server { location / { otel_span_attr http.latency_bucket $latency_bucket; otel_span_attr request.size $request_length; otel_span_attr response.size $bytes_sent; otel_span_attr upstream.addr $upstream_addr; otel_span_attr upstream.response_time $upstream_response_time; proxy_pass http://backend; } } ``` #### 条件属性 ```nginx map $upstream_status $upstream_error { ~^[45] "true"; default "false"; } map $upstream_cache_status $cache_hit { HIT "true"; default "false"; } server { location / { otel_span_attr upstream.error $upstream_error; otel_span_attr cache.hit $cache_hit; otel_span_attr cache.status $upstream_cache_status; proxy_pass http://backend; proxy_cache my_cache; } } ``` ### 业务属性 ```nginx server { location /api/orders { # 业务相关属性 otel_span_attr business.domain orders; otel_span_attr business.criticality high; otel_span_attr business.region $geoip_country_code; # 用户相关属性（注意：避免 PII） otel_span_attr user.type $http_x_user_type; otel_span_attr user.tier $http_x_user_tier; proxy_pass http://order_service; } } ``` ### 使用 Lua 扩展（需要 lua-nginx-module） ```nginx server { location / { access_by_lua_block { local otel = require("opentelemetry") local span = otel.get_current_span() -- 添加自定义属性 span:set_attribute("custom.timestamp", ngx.now()) span:set_attribute("custom.request_hash", ngx.md5(ngx.var.request_uri)) -- 添加事件 span:add_event("request_processing_started", { ["http.method"] = ngx.var.request_method, ["client.ip"] = ngx.var.remote_addr }) } proxy_pass http://backend; } } ``` --- ## 完整配置示例 ### 示例 1: 基础配置 ```nginx # 加载动态模块 load_module modules/ngx_otel_module.so; user nginx; worker_processes auto; error_log /var/log/nginx/error.log notice; pid /var/run/nginx.pid; events { worker_connections 1024; } http { include /etc/nginx/mime.types; default_type application/octet-stream; # OpenTelemetry 导出器配置 otel_exporter { endpoint otel-collector:4317; interval 5s; batch_size 512; batch_count 4; } # 服务标识 otel_service_name nginx-proxy; otel_resource_attr deployment.environment production; otel_resource_attr host.name $hostname; # 全局启用追踪 otel_trace on; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for" ' 'trace_id=$otel_trace_id span_id=$otel_span_id'; access_log /var/log/nginx/access.log main; sendfile on; keepalive_timeout 65; upstream backend { server backend1:8080 weight=5; server backend2:8080 weight=5; keepalive 32; } server { listen 80; server_name localhost; # 健康检查：禁用追踪 location /health { otel_trace off; access_log off; return 200 "healthy\n"; } # 静态资源：采样 location /static/ { otel_trace $http_x_trace_sampled; alias /var/www/static/; expires 1d; } # API 请求：完整追踪 location /api/ { otel_trace on; otel_trace_context propagate; otel_span_name "$request_method $uri"; otel_span_attr http.route /api/*; otel_span_attr api.version v1; otel_span_attr request.id $request_id; proxy_http_version 1.1; proxy_set_header Connection ""; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Request-ID $request_id; # 传递追踪上下文 proxy_set_header traceparent $http_traceparent; proxy_set_header tracestate $http_tracestate; proxy_pass http://backend; } # 默认位置 location / { otel_trace_context inject; proxy_pass http://backend; } } } ``` ### 示例 2: 多环境配置 ```nginx load_module modules/ngx_otel_module.so; events { worker_connections 1024; } http { # 根据环境变量配置 env NGINX_ENV; env OTEL_ENDPOINT; # 动态采样率配置 split_clients "$remote_addr$request_id" $trace_sampled { 10% "1"; * "0"; } map $http_x_b3_sampled $b3_sampled { default ""; "1" "1"; "0" ""; "true" "1"; "false" ""; "d" "1"; } map $b3_sampled$trace_sampled $final_trace { default "0"; ~.*1.* "1"; } # OTLP 导出器 otel_exporter { endpoint ${OTEL_ENDPOINT}; interval 5s; batch_size 512; } otel_service_name nginx-${NGINX_ENV}; otel_resource_attr deployment.environment ${NGINX_ENV}; # 生产环境：按比例采样 # 测试环境：全量采样 otel_trace ${NGINX_ENV} == "prod" ? $final_trace : on; # 上游配置 upstream api_backend { server api1.internal:8080; server api2.internal:8080; } upstream web_backend { server web1.internal:8080; server web2.internal:8080; } # API 网关 server { listen 8080; server_name api.example.com; location / { otel_trace_context propagate; otel_span_name "api:$request_method $uri"; otel_span_attr upstream.service api; otel_span_attr rate.limit.bucket $limit_req_status; proxy_pass http://api_backend; } } # Web 网关 server { listen 80; server_name www.example.com; location / { otel_trace_context inject; otel_span_name "web:$request_method $uri"; otel_span_attr upstream.service web; otel_span_attr cache.status $upstream_cache_status; proxy_pass http://web_backend; proxy_cache web_cache; } } } ``` ### 示例 3: 微服务网关配置 ```nginx load_module modules/ngx_otel_module.so; events { worker_connections 4096; } http { # OpenTelemetry 配置 otel_exporter { endpoint otel-collector:4317; interval 3s; batch_size 256; header X-Scope-OrgID tenant-1; } otel_service_name nginx-microgateway; otel_resource_attr service.namespace platform; otel_resource_attr service.version 1.0.0; otel_resource_attr deployment.environment production; # 追踪配置 otel_trace on; # 日志格式包含追踪信息 log_format trace '$remote_addr - $remote_user [$time_iso8601] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' '"trace_id":"$otel_trace_id",' '"span_id":"$otel_span_id",' '"parent_id":"$otel_parent_id"'; access_log /var/log/nginx/access.log trace; # 服务发现（使用 resolver） resolver 127.0.0.11 valid=30s; # 服务定义 upstream user_service { server user-service:8080 resolve; keepalive 64; } upstream order_service { server order-service:8080 resolve; keepalive 64; } upstream inventory_service { server inventory-service:8080 resolve; keepalive 64; } # 通用追踪配置 map $request_method $trace_operation { GET "read"; POST "create"; PUT "update"; DELETE "delete"; PATCH "patch"; default "unknown"; } server { listen 80; server_name gateway.internal; # 追踪上下文传播 otel_trace_context propagate; # User Service location /api/users/ { otel_span_name "users:$trace_operation"; otel_span_attr service.name user-service; otel_span_attr service.operation $trace_operation; otel_span_attr service.resource users; proxy_pass http://user_service/; proxy_set_header traceparent $http_traceparent; proxy_set_header tracestate $http_tracestate; } # Order Service location /api/orders/ { otel_span_name "orders:$trace_operation"; otel_span_attr service.name order-service; otel_span_attr service.operation $trace_operation; otel_span_attr service.resource orders; proxy_pass http://order_service/; proxy_set_header traceparent $http_traceparent; proxy_set_header tracestate $http_tracestate; } # Inventory Service location /api/inventory/ { otel_span_name "inventory:$trace_operation"; otel_span_attr service.name inventory-service; otel_span_attr service.operation $trace_operation; otel_span_attr service.resource inventory; proxy_pass http://inventory_service/; proxy_set_header traceparent $http_traceparent; proxy_set_header tracestate $http_tracestate; } # 健康检查（无追踪） location /health { otel_trace off; access_log off; return 200 '{"status":"healthy","service":"nginx"}'; } # 追踪信息端点（调试） location /debug/trace { otel_trace on; default_type application/json; return 200 '{ "trace_id": "$otel_trace_id", "span_id": "$otel_span_id", "parent_id": "$otel_parent_id", "sampled": "$otel_parent_sampled" }'; } } } ``` ### 示例 4: Kubernetes 环境配置 ```nginx load_module modules/ngx_otel_module.so; events { worker_connections 1024; } http { # 从环境变量读取 K8s 信息 env KUBERNETES_NAMESPACE; env KUBERNETES_POD_NAME; env KUBERNETES_NODE_NAME; env OTEL_COLLECTOR_SERVICE; # OTLP 导出器 otel_exporter { endpoint ${OTEL_COLLECTOR_SERVICE}:4317; interval 5s; batch_size 512; } # 丰富的资源属性 otel_service_name nginx-ingress; otel_resource_attr k8s.namespace.name ${KUBERNETES_NAMESPACE}; otel_resource_attr k8s.pod.name ${KUBERNETES_POD_NAME}; otel_resource_attr k8s.node.name ${KUBERNETES_NODE_NAME}; otel_resource_attr host.name ${KUBERNETES_POD_NAME}; # 启用追踪 otel_trace on; # 上游配置（K8s Service） resolver kube-dns.kube-system.svc.cluster.local valid=10s; server { listen 80; location / { otel_trace_context propagate; otel_span_name "$request_method $uri"; otel_span_attr k8s.destination.service $proxy_host; otel_span_attr k8s.destination.namespace ${KUBERNETES_NAMESPACE}; # 传递 K8s 相关的追踪头 proxy_set_header X-Request-ID $request_id; proxy_set_header traceparent $http_traceparent; proxy_set_header tracestate $http_tracestate; proxy_pass http://backend-service; } } } ``` --- ## 最佳实践 ### 1. 采样策略 **生产环境建议：** ```nginx # 使用 Head-Based 采样降低开销 split_clients "$request_id" $trace_decision { 5% "1"; # 5% 基础采样 * ""; } # 关键路径始终采样 map $uri $is_critical { default ""; ~*payment "1"; ~*order "1"; ~*auth "1"; } map $trace_decision$is_critical $should_trace { default "0"; ~.*1.* "1"; } otel_trace $should_trace; ``` **关键原则：** - 错误率高的服务：提高采样率 - 高流量服务：降低采样率（0.1% - 1%） - 关键业务路径：全量采样 - 使用 Parent-Based 采样保持追踪链完整 ### 2. 敏感数据处理 **禁止在 Span 属性中包含：** - 密码、API Key - 信用卡号、身份证号 - 个人身份信息 (PII) - 会话令牌 **安全实践：** ```nginx # 正确：使用安全的标识符 otel_span_attr user.id $http_x_user_id; # 用户 ID otel_span_attr session.hash $cookie_session_hash; # 会话哈希 # 错误：不要记录敏感信息 # otel_span_attr user.email $http_x_user_email; # 禁止！ # otel_span_attr auth.token $http_authorization; # 禁止！ # 敏感路径禁用追踪 location /auth/login { otel_span_attr auth.endpoint login; # 不记录请求体 proxy_pass http://auth_service; } ``` ### 3. Span 命名规范 使用清晰、一致的命名： ```nginx # 推荐：包含 HTTP 方法和路径 otel_span_name "$request_method $uri"; # 或按服务分类 otel_span_name "nginx:$request_method $uri"; # 避免：过于笼统或过于详细 # otel_span_name "request"; # 太笼统 # otel_span_name "GET /api/v1/users/12345"; # 包含动态 ID ``` ### 4. 上下文传播 **服务边界处理：** ```nginx # 入口服务：注入新上下文 server { location /api/ { otel_trace_context inject; # 向后传递 proxy_set_header traceparent $http_traceparent; proxy_pass http://backend; } } # 中间服务：传播上下文 server { location / { otel_trace_context propagate; # 既提取上游上下文，又注入到下游 proxy_set_header traceparent $http_traceparent; proxy_pass http://next_service; } } # 出口服务：提取上下文 server { location / { otel_trace_context extract; # 只使用上游传入的上下文，不向后传播 proxy_pass http://final_backend; } } ``` ### 5. 性能优化 **减少开销的配置：** ```nginx http { # 增大批处理大小减少网络开销 otel_exporter { endpoint otel-collector:4317; interval 10s; # 增大导出间隔 batch_size 1024; # 增大批大小 batch_count 8; # 增加队列深度 } # 选择性启用追踪 map $request_uri $trace_enabled { ~*\.(css|js|png|jpg|gif|ico)$ ""; # 静态资源不追踪 /health ""; # 健康检查不追踪 /metrics ""; # 指标端点不追踪 default "1"; # 其他请求追踪 } otel_trace $trace_enabled; } ``` ### 6. 监控 Collector 健康 ```nginx # 监控 OTLP 导出器状态 server { location /nginx_status { stub_status on; allow 10.0.0.0/8; deny all; } location /otel_status { default_type application/json; return 200 '{ "module": "ngx_otel_module", "service_name": "${otel_service_name}", "trace_enabled": "${otel_trace}" }'; } } ``` ### 7. 故障排查 **常见问题及解决方案：** | 问题 | 可能原因 | 解决方案 | |------|----------|----------| | 没有追踪数据 | Collector 不可达 | 检查网络连通性和端口 | | 追踪链断裂 | 上下文传播配置错误 | 检查 `otel_trace_context` 设置 | | Span 名称重复 | 未使用变量 | 使用 `$uri` 或 `$request_uri` | | 采样率异常 | 变量配置错误 | 检查 `split_clients` 或 map | | 属性缺失 | 变量未定义 | 使用 `map` 提供默认值 | **调试配置：** ```nginx # 临时开启详细日志 error_log /var/log/nginx/error.log debug; # 添加调试端点 server { location /debug/otel { default_type application/json; return 200 '{ "trace_id": "$otel_trace_id", "span_id": "$otel_span_id", "parent_id": "$otel_parent_id", "parent_sampled": "$otel_parent_sampled", "request_id": "$request_id", "http_traceparent": "$http_traceparent", "http_tracestate": "$http_tracestate" }'; } } ``` ### 8. 多协议支持 如果后端服务使用不同协议： ```nginx # W3C Trace Context (标准) proxy_set_header traceparent $http_traceparent; proxy_set_header tracestate $http_tracestate; # B3 Propagation (Zipkin) proxy_set_header X-B3-TraceId $otel_trace_id; proxy_set_header X-B3-SpanId $otel_span_id; proxy_set_header X-B3-ParentSpanId $otel_parent_id; proxy_set_header X-B3-Sampled $otel_parent_sampled; # Jaeger Propagation proxy_set_header uber-trace-id "$otel_trace_id:$otel_span_id:$otel_parent_id:$otel_parent_sampled"; ``` --- ## 参考资源 - [NGINX OpenTelemetry Module 官方文档](https://nginx.org/en/docs/ngx_otel_module.html) - [OpenTelemetry 官方文档](https://opentelemetry.io/docs/) - [W3C Trace Context 规范](https://www.w3.org/TR/trace-context/) - [Jaeger 文档](https://www.jaegertracing.io/docs/) - [Zipkin 文档](https://zipkin.io/) --- *文档版本: 1.0 | 最后更新: 2025-01*