Observability Overview

Microtec ERP implements three-pillar observability: Logs, Traces, and Metrics. All observability signals are collected via Serilog and OpenTelemetry and routed to environment-appropriate backends.

---

Three-Pillar Model

┌─────────────────────────────────────────────────────────────┐
│                     Microtec ERP Service                    │
│                                                             │
│  ┌─────────┐    ┌─────────────┐    ┌──────────────────┐   │
│  │ Serilog │    │    OTLP     │    │  Health Checks   │   │
│  │ (Logs)  │    │  (Traces +  │    │  /health endpoint│   │
│  │         │    │   Metrics)  │    │                  │   │
│  └────┬────┘    └──────┬──────┘    └────────┬─────────┘   │
└───────┼─────────────────┼───────────────────┼─────────────┘
        │                 │                   │
        ▼                 ▼                   ▼
   ┌──────────┐    ┌──────────────┐    ┌──────────────┐
   │  Dev:    │    │  Dev:        │    │  Gateway     │
   │  Seq     │    │  Jaeger      │    │  Aggregates  │
   │  :1234   │    │  :16686      │    │  /health     │
   │          │    │  Prometheus  │    │  checks      │
   │  Cloud:  │    │  :9090       │    │              │
   │  App     │    │              │    │  Cloud:      │
   │  Insights│    │  Cloud:      │    │  App Insights│
   └──────────┘    │  App Insights│    │  Availability│
                   └──────────────┘    └──────────────┘

---

Pillar 1: Logs (Serilog)

Technology: Serilog with structured logging

Local Development

• Backend: Serilog → Seq at http://localhost:1234
• Frontend: Angular console logger → Browser DevTools

Cloud Environments

• Backend: Serilog → Application Insights (via Serilog.Sinks.ApplicationInsights)
• Enrichers: Correlation ID (X-Correlation-ID), tenant ID, user ID, service name, environment

See seq-logging.md for detailed Serilog configuration.

---

Pillar 2: Traces (OpenTelemetry)

Technology: OpenTelemetry .NET SDK with OTLP exporter

Local Development

• OTLP HTTP: http://localhost:4318
• OTLP gRPC: http://localhost:4317
• Trace viewer: Jaeger UI at http://localhost:16686

Cloud Environments

• Azure Monitor OpenTelemetry exporter → Application Insights
• Distributed traces available in Application Insights Transaction Search and End-to-End Transaction view

What Is Auto-Instrumented

Library	Traces Generated
ASP.NET Core	Incoming HTTP requests, response codes, duration
EF Core	Database queries with SQL text (dev only)
HttpClient	Outbound HTTP calls with URLs and status codes
Azure Service Bus	Message send/receive operations
Redis (StackExchange)	Cache get/set operations

See opentelemetry.md for configuration details.

---

Pillar 3: Metrics (OpenTelemetry)

Technology: OpenTelemetry Metrics API with OTLP exporter

Local Development

• Prometheus scrape endpoint: /metrics per service
• Prometheus server: http://localhost:9090
• Grafana dashboard: http://localhost:3000

Cloud Environments

• OpenTelemetry metrics → Azure Monitor exporter → Application Insights metrics
• Custom dashboards in Azure Portal workbooks

Default Metrics Collected

Metric	Type	Description
http.server.request.duration	Histogram	Incoming request latency (P50/P95/P99)
http.server.active_requests	UpDownCounter	Concurrent in-flight requests
db.client.operation.duration	Histogram	Database query latency
messaging.process.duration	Histogram	Message processing time
Custom: tenant.active_count	ObservableGauge	Active tenants

---

Health Checks

Per-Service Endpoint

Every microservice exposes /health with readiness and liveness checks:

// In Program.cs
builder.Services.AddHealthChecks()
    .AddSqlServer(connectionString, name: "sql")
    .AddRedis(redisConnectionString, name: "redis")
    .AddAzureServiceBus(connectionString, name: "servicebus")
    .AddCheck<TenantDbHealthCheck>("tenant-db");

app.MapHealthChecks("/health", new HealthCheckOptions
{
    ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
});

Response format:

{
  "status": "Healthy",
  "totalDuration": "00:00:00.0234567",
  "entries": {
    "sql": { "status": "Healthy", "duration": "00:00:00.0123" },
    "redis": { "status": "Healthy", "duration": "00:00:00.0050" },
    "servicebus": { "status": "Healthy", "duration": "00:00:00.0060" }
  }
}

Gateway Aggregation

The Gateway.API aggregates health checks from all downstream services and exposes a combined /health endpoint. This single endpoint is used by:

• Azure Front Door health probes
• Production monitoring alerts
• Pipeline deployment verification

---

Local Development Stack

The full observability stack runs locally via Docker Compose in dev/:

Service	Port	Purpose
Seq	1234	Structured log viewer
Jaeger	16686	Distributed trace viewer
Prometheus	9090	Metrics scrape and query
Grafana	3000	Metrics dashboards
OpenTelemetry Collector	4317, 4318	OTLP receiver and router

Start all observability services:

cd dev
docker compose up -d seq jaeger prometheus grafana otel-collector

---

Cloud Observability Stack

In all cloud environments (dev through production), observability is unified in Azure Monitor:

Signal	Source	Destination
Logs	Serilog HTTP sink	Application Insights
Traces	OTLP → Azure Monitor exporter	Application Insights
Metrics	OTLP → Azure Monitor exporter	Application Insights
Health	/health probes	Application Insights Availability
Container logs	Container Apps runtime	Log Analytics Workspace

---

Correlation and Context Propagation

All signals share the same correlation context using W3C Trace Context (traceparent/tracestate headers):

Client → Gateway (generates trace ID: abc123)
    │ traceparent: 00-abc123-00-01
    ▼
Accounting API (inherits trace ID: abc123)
    │ traceparent: 00-abc123-11-01
    ▼
SQL Database (query tagged with trace ID: abc123)

The X-Correlation-ID header is also propagated as a Serilog log enricher for log-level correlation.

---

Related Pages

• Seq Logging
• OpenTelemetry
• Environments — Application Insights