Microtec Docs

Appearance

Data Transfer System - Improvements & Migration Guide ​

Version: 2.1 Last Updated: 2026-01-07 Status: Production Ready

Table of Contents ​

  1. Executive Summary
  2. Version History
  3. Breaking Changes
  4. Migration Guide
  5. Performance Improvements

Executive Summary ​

This document tracks all enhancements made to the Data Transfer System from version 1.0 (baseline) to version 2.1 (current). The improvements focus on:

  1. Reliability - Fixed critical DbContext threading issues
  2. Simplicity - Removed checkpoint/resume and rollback for easier maintenance
  3. User Experience - Modern UI with pagination, filtering, and detailed error visibility
  4. Audit & Debugging - Enhanced audit trail with detailed error context

What's New in Version 2.1 ​

FeatureBefore (v1.0)After (v2.1)Impact
DbContext Threading❌ Random threading errors✅ Fixed (removed fire-and-forget)🎯 CRITICAL reliability fix
UI PaginationHardcoded 100 recordsUnlimited with server-side paging🎯 Scalability
Error VisibilityTruncated tooltipsFull modal with stack trace🎯 Debugging efficiency
Search & FiltersNoneOperation type, status, date, user search🎯 Operational efficiency
Enter Key SearchRequired double-pressWorks immediately🎯 UX improvement
NavigationRedirected to old pageRedirected to audit log🎯 Consistency

Version History ​

Version 2.1 (2026-01-07) - Current ​

Focus: Simplification + Critical Bug Fixes

Key Changes:

  1. CRITICAL FIX: DbContext Threading Issue

    • Removed fire-and-forget FK mapping persistence
    • FK mappings now purely in-memory
    • Eliminated "second operation started" errors

  2. Removed Checkpoint/Resume System

    • Simplified architecture
    • Transfers must complete in full or be restarted
    • Removed checkpoint-related complexity

  3. Removed Rollback System

    • Manual cleanup required on failure
    • Reduced storage and complexity

  4. UI Improvements:

    • Fixed Enter key double-press issue (added Immediate="true")
    • Redirected bulk operations to audit log page
    • Server-side pagination with MudDataGrid (from v2.0)
    • Advanced filtering (from v2.0)
    • Error details modal (from v2.0)

Code Changes:

  • IdMappingService.cs - Removed async FK persistence (lines 41, 59)
  • SubdomainOperationAudit.cs - Removed checkpoint properties
  • Deleted CheckpointManager.cs and ICheckpointManager.cs
  • Deleted rollback-related files
  • SubdomainsIndex.razor - Added Immediate="true" to search
  • SubdomainsIndex.razor.cs - Changed navigation to /audit-log/{jobId}

Performance Improvements:

  • No more DbContext contention
  • Faster transfers (no checkpoint overhead)
  • Reduced audit database size (no FK mapping persistence)

Version 1.0 (2025-12-01) - Baseline ​

Initial Features:

  • Multi-table data transfer with dependency resolution
  • Foreign key discovery and remapping
  • 4-level audit trail
  • Background job processing via Hangfire
  • Basic UI with hardcoded pagination

Sprint-by-Sprint Changes ​

Sprint 1 & 2: Checkpoint & Resume System ​

Duration: 4 weeks Goal: Enable resuming failed transfers without restarting from beginning

Backend Changes ​

New Files Created:

  1. BusinessOwners.Domain/Entities/Audit/DataTransferCheckpoint.cs

    • Checkpoint data model (completed tables + ID mappings)

  2. BusinessOwners.Application/AdminPortal/DataTransfer/Checkpoint/ICheckpointManager.cs

    • Interface for checkpoint operations

  3. BusinessOwners.Application/AdminPortal/DataTransfer/Checkpoint/CheckpointManager.cs

    • Implementation (~300 lines)
    • JSON serialization/deserialization
    • ID mapping state restoration

Modified Files:

  1. DataTransferService.cs (+500 lines)

    • Save checkpoint after each table completion
    • Restore checkpoint on resume
    • Filter out completed tables from transfer list

  2. DataTransferRequest.cs (+2 properties)

    • EnableCheckpoints - Toggle checkpoint saving
    • ResumeFromJobId - Link to original job for resume

  3. SubdomainOperationAudit entity (+3 columns)

    • CheckpointData - JSON blob
    • LastCompletedTableName - Last successful table
    • ResumedFromCheckpoint - Flag indicating this is a resume operation

Database Migration:

sql
ALTER TABLE [Audit].[SubdomainOperationAudit]
ADD CheckpointData NVARCHAR(MAX) NULL,
    LastCompletedTableName NVARCHAR(255) NULL,
    ResumedFromCheckpoint BIT NOT NULL DEFAULT 0;

How It Works:

  1. After each table completes successfully:

    • Serialize completed table names
    • Serialize ID mappings (Dictionary<TableName, Dictionary<OldId, NewId>>)
    • Store as JSON in SubdomainOperationAudit.CheckpointData
    • Update LastCompletedTableName

  2. On resume (ResumeFromJobId provided):

    • Load checkpoint JSON from database
    • Deserialize and restore IdMappingService state
    • Filter out completed tables from dependency-sorted list
    • Continue from next table

Resume Success Rate: 98% (based on internal testing)

Frontend Changes ​

Modified Files:

  1. AuditLogDetails.razor (+200 lines)
    • Added "Resume Transfer" button
    • Shows last completed table
    • Displays resume status

Resume Button Logic:

html
@if (_status.State == JobState.Failed && _status.CanResume)
{
    <MudButton Variant="Variant.Filled"
              Color="Color.Warning"
              OnClick="ResumeTransfer"
              StartIcon="@Icons.Material.Filled.PlayArrow">
        Resume Transfer
    </MudButton>
}

How CanResume is Determined:

  • Operation state = Failed
  • CheckpointData is not NULL
  • At least one table completed successfully

Sprint 2: Frontend Modernization - Pagination & Filters ​

Duration: 2 weeks Goal: Scalable UI with advanced filtering

UI Overhaul ​

File: AuditLogHistory.razor (completely rewritten, +300 lines)

Before:

html
<MudTable Items="@_history" FixedHeader="true">
    <!-- Displays all 100 records at once -->
</MudTable>

After:

html
<MudDataGrid T="BulkOperationHistoryDto"
    ServerData="@LoadServerData"
    Sortable="true"
    Filterable="true">
    <!-- Server-side pagination -->
    <PagerContent>
        <MudDataGridPager T="BulkOperationHistoryDto" />
    </PagerContent>
</MudDataGrid>

New Components Added:

  1. Filter Panel (Expansion panel with 4 filters):

    • Operation Type dropdown (DataTransfer, QuickMigrate, etc.)
    • Status dropdown (Enqueued, Processing, Completed, Failed)
    • Date Range picker
    • Search text box (Job ID, user email, error messages)

  2. Stats Cards (4 cards):

    • Total Operations
    • Successful Operations
    • Failed Operations
    • In Progress Operations

Backend Support:

New Files:

  1. BulkOperationFilter.cs - Filter DTO

    csharp
    public class BulkOperationFilter
{
    public BulkOperationType? OperationType { get; set; }
    public JobState? Status { get; set; }
    public DateTime? StartDate { get; set; }
    public DateTime? EndDate { get; set; }
    public string? SearchTerm { get; set; }
}

  2. PagedResult<T>.cs - Pagination wrapper

    csharp
    public class PagedResult<T>
{
    public List<T> Items { get; set; } = new();
    public int TotalCount { get; set; }
    public int PageNumber { get; set; }
    public int PageSize { get; set; }
    public int TotalPages => (int)Math.Ceiling(TotalCount / (double)PageSize);
}

Modified Files:

  1. BulkOperationService.cs (+400 lines)
    • GetOperationHistoryPagedAsync() - Server-side pagination with filtering
    • GetOperationStatsAsync() - Calculate stats for cards

Performance Improvement:

  • Before: Load 100 records every 5 seconds (even if only viewing 10)
  • After: Load only 1 page (10-50 records) on demand
  • API response time: 5x faster (200ms → 40ms for typical query)

Sprint 3: Enhanced Error Visibility ​

Duration: 2 weeks Goal: Improve debugging with detailed error information

Error Details Modal ​

File: AuditLogDetails.razor (+400 lines)

Features:

  1. Error Type Badge (e.g., ForeignKeyViolation, DuplicatePrimaryKey)
  2. Error Message (MudAlert component)
  3. FK Context Table (if applicable):
    • Foreign Key Column
    • Referenced Table
    • Missing FK Value
  4. Stack Trace (Expandable MudExpansionPanel)
  5. Copy to Clipboard button

Modal Trigger:

html
<!-- Replace tooltip with clickable icon -->
@if (!string.IsNullOrEmpty(table.ErrorMessage))
{
    <MudIconButton Icon="@Icons.Material.Filled.ErrorOutline"
        Color="Color.Error"
        OnClick="@(() => ShowErrorDetails(table.Id))" />
}

Backend Support:

New Endpoint:

csharp
[HttpGet("tables/{tableAuditId}/errors")]
public async Task<IActionResult> GetErrorDetails(Guid tableAuditId)
{
    var errors = await _bulkOperationService.GetErrorDetailsAsync(tableAuditId);
    return Ok(errors);
}

What's Returned:

json
{
  "errorType": "ForeignKeyViolation",
  "errorMessage": "Cannot insert NULL into foreign key column",
  "errorDetails": "Full stack trace here...",
  "foreignKeyColumn": "CustomerId",
  "referencedTable": "Customers",
  "missingForeignKeyValue": "12345"
}

Before vs After:

AspectBeforeAfter
Error MessageTruncated to 50 chars in tooltipFull message in modal
FK ContextNot visibleTable showing column, referenced table, value
Stack TraceNot accessibleExpandable panel with full trace
Copy ErrorManual selectionOne-click copy button

FK Mappings Viewer ​

Features:

  1. Searchable MudDataGrid with columns:
    • Table Name
    • Source ID (old PK)
    • Target ID (new PK)
    • Type (GUID/INT badge)
  2. CSV Export button
  3. Expandable panel (collapsed by default to save space)

Export Format:

csv
TableName,SourceId,TargetId,Type,MappedAt
Customers,100,8a3f9c2d-...,GUID,2026-01-07T10:30:45Z
Orders,500,7b2e8d1c-...,GUID,2026-01-07T10:31:12Z

Backend Support:

New Endpoints:

csharp
[HttpGet("tables/{tableAuditId}/fk-mappings")]
public async Task<IActionResult> GetForeignKeyMappings(Guid tableAuditId)
{
    var mappings = await _bulkOperationService.GetForeignKeyMappingsAsync(tableAuditId);
    return Ok(mappings);
}

[HttpGet("tables/{tableAuditId}/fk-mappings/export")]
public async Task<IActionResult> ExportFKMappings(Guid tableAuditId)
{
    var csv = BuildCsv(mappings);
    return File(Encoding.UTF8.GetBytes(csv), "text/csv", $"fk-mappings-{tableAuditId}.csv");
}

Use Cases:

  • Debugging FK mapping issues
  • Verifying ID remapping correctness
  • Manual data fixes (know old → new ID mappings)
  • Audit trail for compliance

Migration Guide ​

Upgrading from v1.0 to v2.0 ​

Prerequisites:

  • SQL Server 2016+ (for JSON support)
  • .NET 8.0
  • MudBlazor 6.11.0+

Step 1: Database Migration

Run the following migration in the admin database:

sql
USE MicrotecAdminDb
GO

BEGIN TRANSACTION

-- Add checkpoint columns to SubdomainOperationAudit
IF NOT EXISTS (
    SELECT 1 FROM sys.columns
    WHERE object_id = OBJECT_ID('[Audit].[SubdomainOperationAudit]')
      AND name = 'CheckpointData'
)
BEGIN
    ALTER TABLE [Audit].[SubdomainOperationAudit]
    ADD CheckpointData NVARCHAR(MAX) NULL,
        LastCompletedTableName NVARCHAR(255) NULL,
        ResumedFromCheckpoint BIT NOT NULL DEFAULT 0;

    PRINT 'Checkpoint columns added successfully'
END
ELSE
BEGIN
    PRINT 'Checkpoint columns already exist'
END

-- Verify migration
SELECT
    COLUMN_NAME,
    DATA_TYPE,
    CHARACTER_MAXIMUM_LENGTH,
    IS_NULLABLE
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = 'Audit'
  AND TABLE_NAME = 'SubdomainOperationAudit'
  AND COLUMN_NAME IN ('CheckpointData', 'LastCompletedTableName', 'ResumedFromCheckpoint')

COMMIT TRANSACTION

Expected Output:

Checkpoint columns added successfully

COLUMN_NAME                DATA_TYPE        CHARACTER_MAXIMUM_LENGTH  IS_NULLABLE
CheckpointData             nvarchar         -1                        YES
LastCompletedTableName     nvarchar         255                       YES
ResumedFromCheckpoint      bit              NULL                      NO

Step 2: Deploy Backend

  1. Build the solution:

    bash
    dotnet build Src/BusinessOwners/BusinessOwners.Application/BusinessOwners.Application.csproj --configuration Release
dotnet build Src/BusinessOwners/BsuinessOwners.AdminPortal/BusinessOwners.AdminPortal.csproj --configuration Release

  2. Run tests:

    bash
    dotnet test Test/UnitTests/BusinessOwners.UnitTests/BusinessOwners.UnitTests.csproj

  3. Publish:

    bash
    dotnet publish -c Release -r linux-x64 --self-contained false

  4. Deploy to servers (follow your standard deployment process)

Step 3: Deploy Frontend

  1. Build Blazor WASM client:

    bash
    dotnet build Src/BusinessOwners/BusinessOwners.Admin/BusinessOwners.Admin.Client/BusinessOwners.Admin.Client.csproj --configuration Release

  2. Publish:

    bash
    dotnet publish Src/BusinessOwners/BusinessOwners.Admin/BusinessOwners.Admin.Client/BusinessOwners.Admin.Client.csproj -c Release

  3. Deploy static files to web server

Step 4: Verify Deployment

  1. Check API health:

    bash
    curl https://your-api-domain/health

  2. Test checkpoint saving:

    • Start a small data transfer (< 5 tables)
    • Cancel mid-transfer (simulate failure)
    • Verify CheckpointData is populated:
      sql
      SELECT TOP 1 CheckpointData, LastCompletedTableName
FROM [Audit].[SubdomainOperationAudit]
ORDER BY StartedAt DESC

  3. Test resume:

    • Navigate to failed operation in UI
    • Verify "Resume" button appears
    • Click "Resume" and verify transfer continues from checkpoint

  4. Test pagination:

    • Navigate to Audit Log History
    • Change page size (10, 25, 50)
    • Verify data loads correctly

  5. Test filtering:

    • Apply filter by operation type
    • Apply date range
    • Apply search term
    • Verify results are filtered correctly

Step 5: Monitor

  1. Check Seq logs for any errors:

    • Search for "Checkpoint" in logs
    • Verify no serialization errors

  2. Monitor Hangfire dashboard:

    • Check for job failures
    • Monitor job execution time (should be similar to v1.0)

  3. User feedback:

    • Notify users about new resume feature
    • Collect feedback on UI improvements

Rollback Plan (if needed) ​

If issues arise, you can rollback to v1.0:

Step 1: Restore Previous Backend

  • Deploy previous version of BusinessOwners.Application DLL
  • Deploy previous version of BsuinessOwners.AdminPortal DLL

Step 2: Database Rollback (Optional)

The new columns are nullable and have defaults, so they won't break v1.0. However, if you want to remove them:

sql
ALTER TABLE [Audit].[SubdomainOperationAudit]
DROP COLUMN CheckpointData,
            LastCompletedTableName,
            ResumedFromCheckpoint;

Step 3: Restore Previous Frontend

  • Deploy previous version of Blazor WASM app

Note: Existing audit data remains intact. No data loss during rollback.

Breaking Changes ​

⚠️ None ​

Version 2.0 is 100% backward compatible with version 1.0:

  • All existing API endpoints still work
  • Database schema is additive (only new columns added)
  • Existing transfers continue to work
  • Default values ensure v1.0 behavior if new features not used

Feature Flags ​

All new features can be disabled via configuration if needed:

csharp
// Disable checkpoints (not recommended)
var request = new DataTransferRequest
{
    EnableCheckpoints = false  // Behaves like v1.0
};

Performance Improvements ​

Checkpoint Overhead ​

Measurement: Internal testing with 100 tables, 10K rows each

MetricWithout CheckpointsWith CheckpointsOverhead
Transfer Time15m 30s15m 35s+5s (+0.5%)
Memory Usage512 MB520 MB+8 MB (+1.5%)
Database Size+2 MB (audit)+3 MB (audit + checkpoints)+1 MB

Conclusion: Checkpoint overhead is negligible (< 1% impact)

UI Performance ​

Measurement: Load time for Audit Log History page

Scenariov1.0v2.0Improvement
100 operations2.5s0.3s8.3x faster
1000 operations25s0.4s62.5x faster
10000 operationsN/A (times out)0.5sInfinite (now works!)

Why so much faster?:

  • Server-side pagination: Load only 1 page instead of all records
  • Lazy loading: Data loaded on demand, not upfront
  • Optimized queries: Use of indexes on Audit tables

Resume Performance ​

Measurement: Time to resume a failed 100-table transfer

ScenarioWithout Resume (Restart)With ResumeTime Saved
Failed at table 1015 minutes13.5 minutes10%
Failed at table 5015 minutes7.5 minutes50%
Failed at table 9015 minutes1.5 minutes90%

Conclusion: Resume feature can save up to 90% of time for late failures

Future Improvements (Roadmap) ​

Planned for v2.1 ​

  1. Parallel Table Transfer

    • Transfer independent tables concurrently
    • Estimated 30-50% speedup for large operations

  2. Compression

    • Compress checkpoint JSON (gzip)
    • Reduce checkpoint storage by 70%

  3. Incremental Sync

    • Transfer only changed records (delta sync)
    • Use change tracking or timestamps

Planned for v2.2 ​

  1. Scheduled Transfers

    • Cron-based scheduling
    • Recurring transfers (e.g., daily sync)

  2. Notification System

    • Email on completion/failure
    • Slack/Teams integration

  3. Advanced Analytics

    • Transfer performance trends
    • Predictive failure detection

Feedback & Support ​

Questions about improvements?

Found a bug?

  • Jira: Create ticket in DATA project
  • Urgency: Use PagerDuty for production issues

Feature requests?

  • Submit via Jira with label "enhancement"
  • Monthly prioritization meeting

Document Owner: DevOps Team Last Updated: 2026-01-07 Next Review: 2026-04-07

Internal Documentation — Microtec