Case study: Digital twins, simulation & synthetic data — a pilot that failed (and what it taught us)

Europe's digital twin market reached €7.76 billion in 2024 and is projected to grow at 36.5% CAGR through 2033, yet research from Hong Kong University reveals that 60-70% of digital twin sustainability projects fail to progress beyond pilot stage. The culprit isn't the technology itself—it's the systematic failure to establish proper data governance, realistic KPIs, and what researchers call "model collapse" when synthetic data contaminates training pipelines. This case study dissects a composite European industrial pilot that exemplifies these failure modes, extracting the KPIs that actually matter, benchmark ranges from successful deployments, and the hard lessons that separate pilots that scale from those that quietly disappear.

Why It Matters

Digital twins promise transformative sustainability outcomes: the Bjergmarken wastewater facility in Denmark achieved 30% reduction in N₂O emissions and 25% energy savings through digital twin deployment. Siemens reports 30% production cost reductions in manufacturing implementations. Volkswagen uses digital twins to cut crash test development time by 30%, saving €50 million annually. These results represent the ceiling of what's possible.

The floor, however, is far more common. A 2024 study published in Sustainable Cities and Society identified 45 distinct barriers to digital twin implementation across six clusters, with data uncertainties, legacy system incompatibility, and lack of standardized frameworks topping the list. The European Automobile Manufacturers' Association reports that while EV production supported by digital twins increased 45% over three years, the majority of these implementations remain in pilot or experimental settings rather than routine production.

For European organizations facing the Corporate Sustainability Reporting Directive (CSRD) and EU AI Act compliance requirements, the stakes are particularly high. Digital twins offer a pathway to real-time Scope 1-3 emissions monitoring, life cycle assessment automation, and predictive maintenance that extends asset lifecycles. But failed pilots don't just waste implementation budgets—they create organizational skepticism that delays adoption of technologies that genuinely deliver sustainability outcomes.

The Nature study by Shumailov et al. (2024) documented that AI models trained recursively on synthetic data degrade into "unintelligible blurs" after approximately 30 generations—a finding with direct implications for digital twin simulations that generate synthetic operational data for model training. Understanding these failure modes isn't academic; it's prerequisite knowledge for any organization investing in digital twin infrastructure.

Key Concepts

Digital twins are virtual replicas of physical assets, processes, or systems that receive real-time data from sensors and use simulation to predict performance, optimize operations, and support decision-making. In sustainability contexts, they enable continuous emissions monitoring, energy optimization, and predictive maintenance without physical intervention.

Synthetic data refers to artificially generated datasets that mimic the statistical properties of real operational data. In digital twin applications, synthetic data serves multiple purposes: filling gaps in sensor coverage, augmenting training datasets for machine learning models, and enabling scenario simulation without risking production systems.

Model collapse describes the progressive degradation that occurs when AI models are trained on data generated by previous model generations. Oxford University research demonstrated that as little as 10% contamination of training data with recursively generated synthetic data produces measurable performance degradation. This phenomenon particularly threatens digital twin systems that use synthetic data to train predictive models.

KPI benchmarking in digital twin contexts requires sector-specific calibration. A 78% task completion rate might represent excellence in complex process optimization but mediocrity in predictive maintenance. Without appropriate benchmarks, organizations either underinvest in capable systems or overspend on unnecessary sophistication.

What's Working and What Isn't

What's Working

Structured evaluation before scaling: Organizations achieving top-quartile performance share a common pattern—they implement systematic evaluation on historical data before production deployment. The Bjergmarken wastewater facility built pilot studies with live sensor data before full deployment, validating digital twin accuracy against known operational outcomes. This investment typically represents 15-25% of initial development time but prevents costly post-deployment remediation.

Hybrid architectures with appropriate automation levels: Pure autonomy rarely works in sustainability applications. The best-performing deployments use tiered approaches: fully autonomous handling for high-confidence, low-stakes decisions (routine energy optimization); human-in-the-loop for medium-confidence or high-stakes decisions (emissions reporting); human-on-the-loop for monitoring aggregate patterns (regulatory compliance). Microsoft's internal deployment data shows hybrid architectures achieve 23% higher user satisfaction than fully autonomous alternatives.

Observability-first design: Top performers instrument their digital twins heavily from day one. This means logging not just inputs and outputs, but reasoning traces, confidence scores, sensor data quality metrics, and decision points. The median enterprise deployment now generates 2-5 MB of logs per operational hour, a 4x increase from 2023 norms.

Accumulation rather than replacement data strategies: The 2024 research breakthrough from Gerstgrasser et al. demonstrated that accumulating synthetic data alongside real data prevents model collapse, while replacement strategies cause progressive degradation. Organizations implementing "accumulate" approaches maintain stable or improving model performance over 40+ training iterations.

What Isn't Working

Technology-first implementation without data strategy: Research published in the Journal of Cleaner Production found that projects jumping to implementation without establishing data pipelines, quality controls, or governance frameworks fail at significantly higher rates. The prerequisite is comprehensive energy and resource audits to establish baseline metrics before building the digital twin.

Vanity metrics masking real performance: Many organizations report impressive KPIs that collapse under scrutiny. Common problems include counting partial completions as successes, measuring task completion without verification, and using model self-reports rather than ground truth. One manufacturing case study discovered their reported 89% accuracy was actually 61% when systematic sampling was implemented.

Ignoring distribution tails and edge cases: Digital twins often achieve 95%+ accuracy on common operational scenarios but catastrophically fail on rare events—precisely the scenarios where sustainability impact is often highest (equipment failures, extreme weather, supply chain disruptions). These tail failures, comprising 2-5% of operational cases, can destroy more value than the other 95% creates.

Premature scaling based on pilot success: Pilot conditions differ fundamentally from production: curated data types, motivated operators, extra attention from developers. Production environments surface failure modes that pilots miss. Industry analysis found that 67% of scaled deployments underperformed their pilots by at least 20% on key metrics.

Regulatory gray areas: The EU Medical Device Regulation (MDR 2017/745) and EU AI Act (2024) create compliance complexity for digital twins classified as high-risk AI systems. Organizations deploying digital twins for sustainability reporting without clarifying regulatory classification face potential enforcement actions and liability exposure.

Key Players

Established Leaders

Siemens (Germany) — Market leader with 46% share in hardware solutions and the Xcelerator platform. The company's January 2025 partnership with NVIDIA launched the Teamcenter Digital Reality Viewer, integrating industrial digital twins with AI-powered visualization. Revenue of €75.9 billion in FY2024 and 312,000+ employees make Siemens the dominant European player.

Dassault Systèmes (France) — The 3DEXPERIENCE platform powers digital twins across aerospace, automotive, and manufacturing. Key tools include CATIA for design and DELMIA for manufacturing workflow simulation. Notable implementations include Hyundai Heavy Industries LNG carrier digital twins and Jaguar Land Rover workflow optimization.

ABB (Switzerland) — Industrial automation leader with digital twin solutions spanning robotics, electrification, and process industries. The company's focus on energy efficiency and emissions monitoring aligns with European sustainability requirements.

Schneider Electric (France) — Invested €1 billion in smart city digital twin solutions in November 2023, with particular focus on building energy management and grid optimization across European infrastructure.

Emerging Startups

Gradyent (Netherlands) — Raised €28 million Series B in 2024 for energy grid optimization using digital twins. The company's technology enables district heating networks to reduce energy losses and carbon emissions through predictive modeling.

OroraTech (Germany) — Secured €37 million Series B in 2024 for wildfire monitoring via satellite-based digital twins. The platform provides real-time fire detection and spread prediction for forestry and infrastructure protection across Europe.

Syntho (Netherlands) — AI platform generating synthetic data twins for secure test environments, addressing GDPR compliance challenges that complicate European digital twin deployments.

Tomorrow Things (Germany) — Raised €1.5 million in March 2024 for AI-driven digital twin intelligence for industrial assets, focusing on predictive maintenance and operational optimization.

Key Investors & Funders

EU Horizon Europe — Allocated €250 million for digital twin projects focused on energy and manufacturing applications, with additional €80 million for healthcare digital twins through the Virtual Human Twins Initiative.

High-Tech Gründerfonds (Germany) — Leading early-stage investor in European digital twin startups, with particular focus on industrial applications.

Earlybird Venture Capital (Germany) — Active investor in deeptech including digital twin and synthetic data platforms across the European startup ecosystem.

UK Digital Twin Centre — £37.6 million initiative with participation from Thales and Spirit AeroSystems, focused on advancing digital twin capabilities for aerospace and defense applications.

Examples

Bjergmarken Wastewater Treatment Plant (Denmark): This facility deployed a comprehensive digital twin modeling biological processes, energy balances, and dynamic CO₂ footprint calculations. The implementation achieved 30% reduction in N₂O emissions (which have 273x the warming potential of CO₂), 25% energy savings, and maintained effluent quality standards. Critical success factor: the team built the digital twin on verified process models with continuous sensor validation, avoiding the synthetic data contamination that degrades predictive accuracy. KPIs tracked included energy balances, CO₂ footprint, effluent quality, and operational parameters with real-time dashboards.

German Automotive Consortium (Failed Pilot): A consortium of three German automotive manufacturers attempted to create a shared digital twin platform for Scope 3 supply chain emissions tracking in 2024. The pilot failed after 14 months due to data standardization conflicts between legacy ERP systems, inability to verify synthetic data quality from tier-2 suppliers, and model collapse in the predictive emissions algorithms after recursive training on supplier-reported data. Lessons learned: the consortium attempted a "big bang" deployment without establishing common data formats, governance frameworks, or verification mechanisms for synthetic data. Post-mortem analysis revealed that 23% of supplier data was machine-generated without human verification, contaminating the training pipeline.

Matera Urban Digital Twin (Italy): This city-wide implementation for urban planning, infrastructure maintenance, and tourism management demonstrates both successes and ongoing challenges. The platform integrates multiple systems including energy, water, and transportation networks. While operational monitoring functions effectively, the project highlights the coordination complexity required for multi-disciplinary digital twins—the team identified stakeholder alignment and governance gaps as primary obstacles to moving from pilot to production scale across all city services.

Action Checklist

• Conduct comprehensive baseline audit of physical systems (energy, water, emissions) before initiating digital twin development—this data establishes ground truth for validation
• Define sector-appropriate KPIs with explicit success criteria before deployment, not after; align metrics with business objectives using frameworks like ISO 14067 for carbon footprint calculations
• Establish data governance framework including quality controls, provenance tracking, and verification mechanisms for any synthetic data entering training pipelines
• Implement "accumulate" strategy for training data: mix real sensor data with synthetic augmentation rather than replacing real data with synthetic alternatives
• Build observability infrastructure concurrent with digital twin development; instrument for logging inputs, outputs, confidence scores, and decision points
• Design graceful degradation paths for each identified failure mode—digital twins should fail safely with human escalation rather than silently or catastrophically
• Run controlled pilots with systematic comparison to baseline performance for minimum 6 months before scaling; budget 15-25% of development time for structured evaluation
• Clarify regulatory classification under EU AI Act and relevant sector regulations before production deployment; document compliance approach for high-risk applications
• Establish sampling-based verification (minimum 5% of operational decisions) to catch silent failures and accuracy degradation over time
• Create feedback loops from failure cases back to model improvement; tail cases and edge failures often reveal the highest-impact optimization opportunities

FAQ

Q: What KPIs should we prioritize for a sustainability-focused digital twin deployment?

A: Core metrics should span four categories: environmental impact (energy consumption in kWh, carbon footprint in CO₂e using ISO 14067 factors, water intensity per unit), operational efficiency (equipment uptime via MTBF/MTTR, first-pass accuracy with 72-hour verification window), financial performance (cost per optimized decision, ROI timeline targeting positive returns within 2-3 years), and model health (prediction accuracy vs. ground truth, synthetic data ratio in training pipeline, model drift indicators). Top performers track 15-20 specific metrics but focus governance attention on 4-6 that most directly link to sustainability outcomes. Avoid vanity metrics—a high task completion rate is meaningless without verification that "completed" tasks actually achieved their intended outcomes.

Q: How do we prevent model collapse when using synthetic data to augment our digital twin training?

A: The 2024 research from Oxford University and others established three key strategies. First, implement the "accumulate" approach: add synthetic data to real data rather than replacing real data. Studies show this prevents collapse over 40+ training iterations. Second, cap synthetic data ratios—set maximum percentages (typically 20-40% depending on domain) and track provenance to ensure real data remains the majority. Third, deploy verification filtering: use separate quality assessment models to evaluate synthetic data before it enters training pipelines. The FCA UK report recommends structured value-risk assessment for any synthetic data application in regulated industries. Finally, maintain repositories of verified, pre-contamination data from before synthetic generation became widespread—this provides clean baseline data for model recovery if degradation is detected.

Q: What's a realistic timeline from pilot to production-scale digital twin deployment?

A: Based on 2024-2025 European implementations, expect a staged journey. Foundation phase (months 1-6): baseline audits, KPI definition, data strategy, stakeholder training. Basic twin phase (months 6-12): IoT sensor deployment, virtual model construction, data pipeline establishment, baseline monitoring. Optimization phase (months 12-24): predictive analytics, scenario simulation, closed-loop control for low-risk decisions. AI-driven phase (months 24+): autonomous decision-making, system-of-systems integration, continuous improvement cycles. Most organizations see rapid initial improvement in weeks 1-4, plateau during structured evaluation in weeks 5-12, then gradual optimization through months 3-12. Rushing this timeline typically results in the premature scaling failures documented in 67% of deployments that underperform pilots.

Q: How do European regulatory requirements (CSRD, EU AI Act) affect digital twin deployment?

A: The EU AI Act (2024) creates classification requirements for digital twins depending on their application. Systems making decisions affecting environmental compliance, worker safety, or critical infrastructure may fall under high-risk categories requiring conformity assessments, documentation, and human oversight provisions. CSRD mandates comprehensive sustainability reporting including Scope 3 emissions—digital twins can automate much of this data collection and calculation, but the outputs must meet audit-grade verification standards. Organizations should: (1) conduct preliminary AI Act classification during project planning, (2) document the decision-making role of digital twin systems, (3) implement human oversight mechanisms for high-stakes environmental decisions, and (4) ensure audit trails support CSRD disclosure requirements. Regulatory gray areas remain, particularly for digital twins that don't neatly fit existing classification frameworks—early engagement with legal counsel and potentially with national supervisory authorities reduces compliance risk.

Q: Should we build custom digital twin infrastructure or use established platforms like Siemens Xcelerator or Dassault 3DEXPERIENCE?

A: Platform selection depends on three factors: integration complexity, customization requirements, and internal capabilities. For organizations with standard industrial assets and established sensor infrastructure, platforms like Siemens Xcelerator or Dassault 3DEXPERIENCE offer faster time-to-value, proven reliability, and ongoing support—Siemens reports that platform-based implementations typically reach production 40% faster than custom builds. Custom development makes sense when: (1) existing platforms don't support your specific asset types or data formats, (2) proprietary algorithms provide competitive advantage, or (3) regulatory requirements mandate specific data handling approaches. Many successful deployments use hybrid approaches: platform foundations with custom analytics layers. Budget 20-30% more for integration when combining platforms with legacy systems—this is where most hidden costs emerge. European startups like Gradyent and OroraTech offer specialized solutions that bridge platform and custom approaches for specific domains.

Sources

• Shumailov, I., Shumaylov, Z., Zhao, Y., et al. (2024). AI models collapse when trained on recursively generated data. Nature, 631(8022), 755-759.
• Sustainable Cities and Society (2024). Modelling the relationship between digital twins implementation barriers and sustainability pillars: Insights from building and construction sector. ScienceDirect.
• Global Market Insights (2024). Digital Twin Market Size & Share, Growth Analysis 2025-2034. Market Research Report.
• DHI Group (2024). When wastewater becomes smart: Digital twins open new doors. Technical Case Study, Bjergmarken Wastewater Treatment Plant.
• Financial Conduct Authority UK (2025). Using synthetic data in financial services. Regulatory Report.
• Gerstgrasser, M., et al. (2024). Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data. ICLR 2024.
• Market Data Forecast (2024). Europe Digital Twin Market Size, Share, 2033. Regional Market Analysis.
• BuildingSmart International (2024). Digital Twins and the Systems Perspective. Technical Whitepaper.