Microtec ERP — Azure Container Apps Audit Report

Updated: 2026-03-26  |  Scope: Platforms, InfrastructureServices, WorkflowDesigner, Microtec HR (Keycloak excluded)  |  Services: 13 microservices + 1 API gateway

Executive Summary

8
Critical
5
Warning
4
Prod Ready
1
New Feature
CategoryCriticalWarning
1. Hangfire & Background Jobs52
2. File I/O & Ephemeral Storage32
3. Caching & State—1
4. New Feature: Multi-Tenant Connection Dictionary——
TOTAL85

Service Readiness Matrix

Ready = Ready for Production

#ServiceRepoHangfireFile I/OCache/StateOverall
1gateway-apiPlatforms——✅✅ PROD
2bo-apisPlatformsEnqueue—✅❌
3adminportal-apisPlatformsRecurring+EnqueueVolume mounts✅❌
4appsportal-apisPlatformsRecurring+Outbox—✅❌
5inventory-apisPlatformsDashboard—✅⚠️
6integration-apisPlatformsWebhooks—✅❌
7attachment-apisInfraServicesRecurring cleanup—✅❌
8notification-apisInfraServices——✅✅ PROD
9workflows-apisWorkflowDesigner——✅✅ PROD
10hr-apisMicrotec HR——✅✅ PROD

✅ Ready for Production   ⚠️ Minor fix needed   🔴 Critical fix needed   — Not applicable

Development Effort & Timeline

All estimates are for 1 developer. Rows are ordered by recommended execution sequence.

# Work Item Category Services Affected Est. Days
1 Refactor background jobs — system-wide architecture
Design & implement custom job entry point, remove Hangfire dependency, wire up ACA Jobs / Service Bus pattern
⚙️ Requires a Software Engineer		 Hangfire All 6 Hangfire services 2 days
2 Apply bg job refactor in bo-apis
Replace BackgroundJob.Enqueue calls (backup, bulk) with new entry point		 Hangfire bo-apis 2 days
3 Apply bg job refactor in appsportal + inventory + attachment
Migrate recurring outbox, notifications, file cleanup jobs; remove Hangfire dashboard		 Hangfire appsportal-apis, inventory-apis, attachment-apis TBD
4 Fix connection string issue — Multi-Tenant Server Registry
Replace encrypted per-tenant connection strings with ServerId + DatabaseName pattern; migrate tenant table		 New Feature bo-apis, adminportal-apis, appsportal-apis, inventory-apis 2 days
5 Fix AdminPortal File I/O issue
Mount Azure File Share for translation backups, SQL scripts, and cache directory (FIO-1, FIO-2, FIO-3)		 File I/O adminportal-apis 1 day
6 Integrate adminportal with App Insights
Wire up Application Insights SDK, configure telemetry, link to existing workspace		 Observability adminportal-apis 1 day
7 Test & validate App Insights logs
Verify traces, exceptions, dependencies, and custom events flow correctly across all services in the portal		 Observability adminportal-apis 1 day
8 Integrate Redis & health checks in adminportal
Replace local filesystem cache with Redis distributed cache; add /health endpoint covering DB, Redis, and storage dependencies		 Cache / Health adminportal-apis 1 day
Total (known estimates) 10+ days

Category 1: Hangfire & Background Jobs

🔴 HF-1: Recurring Jobs Duplicate on Multiple Replicas
Impact: Data corruption, duplicate notifications, duplicate backups
Root Cause: RecurringJob.AddOrUpdate() registers jobs on every replica. All replicas compete to execute the same job simultaneously.

5 Recurring Jobs Affected:

HF-1a: Translation Backup — Daily Midnight

Platforms/Src/BusinessOwners/BsuinessOwners.AdminPortal/DependencyInjection.cs:11-14

RecurringJob.AddOrUpdate<IBackupService>(
    nameof(BackupService),
    job => job.BackupTranslationFileAsync(),
    CronKeys.EveryDayAtMidnight);

With 3 replicas: 3 backups created simultaneously, writing to same path → file corruption.

HF-1b: Outbox Message Processing — Daily Midnight

Platforms/Src/AppsPortal/Accounting/AppsPortal.Application/DependencyInjection.cs:34-37

RecurringJob.AddOrUpdate<IOutboxMessageBackGroundJob>(
    nameof(OutboxMessageBackGroundJob),
    job => job.ExecuteAsync(),
    CronKeys.EveryDayAtMidnight);

With 3 replicas: Same domain events published 3 times → downstream systems receive duplicate messages.

HF-1c: Notification Processing — Dynamic Frequency

AppsPortal.Application/.../ActivateNotificationSettingCommandHandler.cs:33-55

RecurringJob.AddOrUpdate<INotificationProcessingJob>(
    jobId,
    job => job.ExecuteAsync(settingType, tenantContext),
    cronExpression);

With 3 replicas: Users receive 3x duplicate notifications. Dynamic registration creates race conditions between replicas calling RemoveIfExists + AddOrUpdate.

HF-1d: Notification Settings Update — Dynamic

AppsPortal.Application/.../UpdateNotificationSettingCommandHandler.cs:63-75

Same pattern as HF-1c — race condition between RemoveIfExists and AddOrUpdate across replicas.

HF-1e: File Cleanup — Daily Midnight

InfrastructureServices/Attachments/Attachment.Apis/DependencyInjection.cs:24-27

RecurringJob.AddOrUpdate<IFileCleanupService>(
    nameof(FileCleanupService),
    job => job.CleanUpUnusedFiles(),
    CronKeys.EveryDayAtMidnight);

With 3 replicas: 3 pods attempt to delete the same files simultaneously → race conditions, exceptions, partial cleanup.

🔴 HF-2: Scale-to-Zero Kills Hangfire Workers
Impact: Missed scheduled jobs, orphaned job records
Services: All 6 Hangfire-enabled services (BO, AdminPortal, AppsPortal, Inventory, Integration, Attachment)

When KEDA scales containers to 0 replicas:

🔴 HF-3: Fire-and-Forget Jobs — No Idempotency
Impact: Duplicate execution if pod crashes mid-job

6 Fire-and-Forget Enqueue Locations:

HF-3a: Database Backup

BusinessOwners.Application/AdminPortal/DatabaseBackup/DatabaseBackupService.cs:36

BackgroundJob.Enqueue<DatabaseBackupBackgroundJob>(job =>
    job.ExecuteAsync(jobId, request));

HF-3b: Bulk Operations

BusinessOwners.Application/AdminPortal/BulkOperation/BulkOperationService.cs:36

BackgroundJob.Enqueue<BulkOperationBackgroundJob>(job =>
    job.ExecuteAsync(jobId, request));

HF-3c: Webhook Delivery

Integration.Application/WebhooksManager/Services/WebhookDeliveryService.cs

BackgroundJob.Enqueue<IWebhookDeliveryService>(service =>
    service.ProcessDeliveryAsync(subDomainName, deliveryId));
BackgroundJob.Schedule<IWebhookDeliveryService>(
    service => service.ProcessDeliveryAsync(subDomainName, deliveryId),
    delay);

HF-3d: Hangfire Background Job Service (Generic)

AppsPortal.Application/Sales/SyncBack/.../HangfireBackgroundJobService.cs:8-26

public void Enqueue<TJob>(Expression<Func<TJob, Task>> methodCall) =>
    BackgroundJob.Enqueue(methodCall);

Generic wrapper — all callers inherit the same vulnerability.

HF-3e: ZATCA Invoice Processing

AppsPortal.Application/Sales/.../ZatcaBackgroundJob.cs:160

BackgroundJob.Schedule<ZatcaBackgroundJob>(
    job => job.RetryFailedInvoicesAsync(subDomainName, tenantContext),
    TimeSpan.FromMinutes(5));

HF-3f: Data Transfer Job

BusinessOwners.Application/AdminPortal/DataTransfer/.../DataTransferJobService.cs:48

Uses IBackgroundJobClient (DI-injected — better pattern but same idempotency concern).

🔴 HF-4: Distributed Lock Failures with DisableConcurrentExecution
Impact: Multiple replicas execute "locked" jobs simultaneously
Files: WebhookDeliveryService.cs — [DisableConcurrentExecution(300)] · ZatcaBackgroundJob.cs — [DisableConcurrentExecution(600)]

These Hangfire attributes use SQL-based distributed locks. With multiple Hangfire servers (replicas), lock contention causes:

🔴 HF-5: Remove Hangfire — Create Custom Entry Point for Job Execution
Impact: Hangfire adds complexity, connection overhead, and multi-replica issues incompatible with ACA scaling
Services: 6 Hangfire-enabled services (BO, AdminPortal, AppsPortal, Inventory, Integration, Attachment)

Hangfire should be fully removed. Replace with:

Pattern:

// JobsController.cs — custom entry point replacing Hangfire
[ApiController]
[Route("jobs")]
[ApiKey] // Internal API key auth only
public class JobsController(IServiceProvider services) : ControllerBase
{
    [HttpPost("{jobName}")]
    public async Task<IActionResult> Execute(string jobName, [FromBody] JsonElement? parameters)
    {
        var job = services.GetKeyedService<IJob>(jobName);
        if (job is null) return NotFound();
        await job.ExecuteAsync(parameters);
        return Ok();
    }
}

This eliminates Hangfire's SQL polling, dashboard exposure, distributed lock issues, and connection pool overhead.

⚠️ HF-6: Static API Usage
Impact: Harder to test, mock, or swap implementations. Must be replaced when migrating away from Hangfire.

All static BackgroundJob.Enqueue() and RecurringJob.AddOrUpdate() calls need to be refactored to use the custom job entry point (see HF-5) or Service Bus queue dispatch.

⚠️ HF-7: BackgroundPerformanceLogger Data Loss
AppsPortal.Infrastructure/Services/Performance/BackgroundPerformanceLogger.cs
Registered in: AppsPortal, Inventory, Integration (3 services)

Uses in-memory Channel<PerformanceLogEntry> to batch logs every 5 seconds. Scale-to-zero or restart drops all buffered logs. BoundedChannelOptions with DropOldest silently discards data under load.

Category 2: File I/O & Ephemeral Storage

🔴 FIO-1: Translation Backups Write to wwwroot
BusinessOwners.Application/AdminPortal/TranslationBackups/BackupService.cs
Lines: 14, 48, 56, 71, 75, 78, 109, 118
// Line 14 — hardcoded to wwwroot
Path.Combine("wwwroot", _configuration[ConfigKeys.BackupsUploadPath])

// Line 75-78 — creates directory and writes backup SQL to disk
Directory.CreateDirectory(directoryPath);
using var writer = new StreamWriter(filePath, ...);

Impact: Translation backup files lost on every container restart. Database records reference non-existent files.

Fix: Attach an Azure File Share volume to the container app and mount it at the wwwroot backup path. No code changes needed.
🔴 FIO-2: SQL Script Persistence to Filesystem
BusinessOwners.Application/AdminPortal/SqlScripts/SqlScriptService.cs
Lines: 24-26, 30, 37, 45, 75, 90, 103
// Line 24-26 — resolves script file path
Path.Combine(AppContext.BaseDirectory, options.Value.ScriptsFilePath)

// Line 75, 90, 103 — writes updated scripts back to disk
WriteScriptsToFileAsync(...)

Impact: If SQL scripts are created/edited at runtime, all changes lost on restart.

Fix: Attach an Azure File Share volume and mount it at the scripts path. No code changes needed.
🔴 FIO-3: JSON Command Configuration File (if writable)
BusinessOwners.Application/AdminPortal/Commands/Services/JsonCommandService.cs
Lines: 15-18, 30, 36, 88, 94
Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "AdminPortal/Commands/Data/commands.json")

Impact: If commands.json is baked into Docker image and read-only → OK. If modified at runtime → changes lost.

⚠️ FIO-4: Cache Directory Operations
AdminPortal/Commands/Handlers/Json/JsonCommandHandler.cs
AdminPortal/Commands/Handlers/System/ClearCacheCommandHandler.cs

Creates/clears Cache/ directory relative to app root. Acceptable for transient caching — but cache is not shared across replicas.

⚠️ FIO-5: AzureFileService Temp File Leak
Attachment.Application/FileService/AzureFileService.cs:207-222
var tempFilePath = Path.GetTempFileName();
var download = fileClient.Download();
using (var fileStream = File.Create(tempFilePath))
{
    download.Value.Content.CopyTo(fileStream);
}
return new FileStream(tempFilePath, FileMode.Open, FileAccess.Read,
    FileShare.None, 4096, FileOptions.DeleteOnClose);

Creates temp file for streaming downloads. DeleteOnClose flag handles cleanup, but if exception occurs between creation and FileStream construction, temp file leaks. Minor issue.

Category 3: Caching & State

⚠️ CS-1: Cache Directory on Local Filesystem

Files in AdminPortal/Commands/Handlers/ create/clear Cache/ directory on local filesystem. Cache not shared between replicas. Acceptable for non-critical caching, but should use Redis for consistency.

Category 4: New Feature — Multi-Tenant Connection Dictionary

Affected: AppsPortal, Inventory

Current Pattern

Each tenant stores a full encrypted connection string in the database — server URL, credentials, and database name all baked together. Changing the SQL server requires re-encrypting every tenant's connection string.

Proposed Pattern — Server Registry + DB Name per Tenant

Config (appsettings / Key Vault) — Define SQL servers as a named registry:

{
  "MultiTenantsServer": [
    {
      "Id": "ServerA",
      "Url": "10.100.1.4",
      "Port": 1433,
      "User": "sa",
      "Password": "{{from-key-vault}}"
    },
    {
      "Id": "ServerB",
      "Url": "10.200.1.4",
      "Port": 1433,
      "User": "sa",
      "Password": "{{from-key-vault}}"
    }
  ]
}

Database (Tenant table) — Store only ServerId + DatabaseName per tenant:

SubDomainServerIdDatabaseName
testServerAMicrotecApps.shared
testdoServerAMicrotecApps.TestDo
client-xServerBMicrotecApps.ClientX

Connection string built at runtime:

public class TenantInfo
{
    public string SubDomain { get; set; }
    public string ServerId { get; set; }      // e.g. "ServerA"
    public string DatabaseName { get; set; }  // e.g. "MicrotecApps.shared"
}

// ConnectionStringBuilder resolves ServerId → server config, appends DatabaseName
var connStr = connectionStringBuilder.Build(tenant.ServerId, tenant.DatabaseName);
// → "Server=10.100.1.4,1433;Database=MicrotecApps.shared;User Id=sa;Password=...;"

Implementation

  1. Add MultiTenantsServer config section with server registry array
  2. Add ServerId column to tenant table (migrate existing tenants — all currently point to same server)
  3. Create ConnectionStringBuilder service that resolves ServerId → server config + appends DatabaseName
  4. Replace encrypted connection string column with plain DatabaseName column
  5. Update ITenantProvider / ITenantContextManager to use the builder
  6. Key Vault: store server credentials per server ID (MultiTenantsServer--0--Password, etc.)

Benefits

Appendix: Hangfire Services Configuration

ServiceAddHangfireUseHangfireServerUseHangfireDashboardRecurring JobsEnqueue
BO APIs✅✅——✅ (backup, bulk)
AdminPortal✅✅✅✅ (translation backup)✅ (backup, bulk, data transfer)
AppsPortal✅✅✅✅ (outbox, notifications)✅ (sync, zatca, generic)
Inventory✅✅✅——
Integration✅✅✅—✅ (webhooks)
Attachment✅✅✅✅ (file cleanup)—
Notifications—————
Workflows—————
HR—————
Gateway—————

End of Report