Best fit
- Microsoft Fabric
- Fast governed analytics, Power BI and lower operating overhead
- Databricks
- Advanced ML, high-volume telemetry and engineering flexibility
Target design · TD-001 · proposed
Deployable technical architecture
A lakehouse plus warehouse serving certified measures, operational APIs and permission-aware retrieval. All paths are environment-separated, identity-controlled, observable and replayable.
Flow
00
Sources
01
Connectors
02
Bronze Delta
03
Silver products
04
Semantic models
05
AI / BI serving
00 — Sources
16 registered applications; extraction contracts pending validation
DMAHEATiPortalGIS PortalSCADA ApplicationNFA PortalHSE iReporting PortalLoyalty PortalQAM PortalSupplier PortalDBMS / LMS PortalHRMSCustomer Self-Service PortalSAP (ERP)TIPMSpot Billing
01 — Ingestion
Incremental extraction and contract enforcement
- CDC (Debezium) on MariaDB / MySQL / PostgreSQL sources
- SAP SLT + OData extractors for ERP objects
- MQTT / OPC-UA bridge for SCADA telemetry
- REST + webhook connectors for .NET and Laravel portals
- Schema registry with contract validation at the edge
02 — Data Lake (Bronze)
Immutable Delta landing and replay
- Object storage with open table format (Iceberg / Delta)
- Partitioned by source system and event date
- Exactly-once landing with checkpoint offsets
- Raw payload retained with ingest metadata columns
03 — Conformed (Silver)
Conformed entities, history and quality controls
- Deduplication, type casting, late-arriving record handling
- Master data resolution: customer, premise, asset, employee, supplier
- GIS geometry standardisation (single CRS) for spatial joins
- Data quality tests (freshness, uniqueness, referential integrity)
04 — Warehouse (Gold)
Certified semantic measures and governed SQL
- Dimensional models: dim_customer, dim_asset, dim_geo, dim_employee
- Facts: fct_connection_funnel, fct_consumption, fct_incident, fct_billing
- Semantic/metric layer so every tool shares one definition
- Incremental builds orchestrated with dbt + Airflow
05 — AI & Consumption
Policy-aware retrieval and monitored model serving
- BI layer on the semantic models
- Feature store for ML training and serving
- Vector store over documents (HSE, policies, SOPs) for RAG
- Text-to-SQL assistant scoped to governed gold models
Non-functional control plane
Controls applied to every environment
Access & privacy
Entra groups, least privilege, managed identities, secrets vault, row/column security, masking and immutable audit.
Catalogue & lineage
Owner, classification, schema, quality SLO and column lineage required before certification.
Cost & reliability
Freshness, completeness, duplicates, drift, job failures, model quality, token and capacity consumption.
Release & recovery
Dev/test/prod isolation, IaC, pull-request gates, deployment identity, rollback, backup and timed restore exercise.
Decision record · benchmark required
One primary platform; earn any second platform
No final selection is valid before identical SAP/GIS workloads are benchmarked for elapsed time, source impact, concurrency, operability, security controls and total capacity. Pricing is excluded until measured volumes and commercial terms are available.
Storage & BI
- Microsoft Fabric
- OneLake, Delta and native DirectLake semantic models
- Databricks
- Open Delta lakehouse with broad BI connectivity
Data science
- Microsoft Fabric
- Integrated and improving; enough for early models
- Databricks
- Mature MLflow, feature engineering and model serving
Governance
- Microsoft Fabric
- Purview plus Microsoft identity and policy ecosystem
- Databricks
- Unity Catalog with strong fine-grained lineage and controls
Selection gate
- Microsoft Fabric
- Choose if benchmark meets SLOs and Microsoft support model is viable
- Databricks
- Choose if telemetry/ML/geospatial benchmark materially outperforms and skills exist
Company Brain query path
- 01 · Authenticate user and resolve role, geography and purpose.
- 02 · Route metric questions to certified semantic models; documents to hybrid retrieval.
- 03 · Apply row/column policy before retrieval or SQL execution.
- 04 · Return citations, freshness and metric definition; abstain below confidence threshold.
- 05 · Log prompt, sources, policy result, response and user feedback.
Recovery objectives · proposed
- Critical serving
- RTO 4h / RPO 1h — validate
- Analytical products
- RTO 8h / RPO 24h — validate
- Pipeline failure
- Quarantine; alert owner; replay from durable checkpoint
- AI failure
- Disable endpoint; retain governed report/manual workflow