🔧 Implement & 🐳 deployment
Strategies and Processes
This subsection outlines the specific methods and steps for integrating and deploying the DIDImo solution, emphasizing its evolution through a user-centric approach. The architecture design was refined significantly based on feedback gathered through detailed questionnaires and user interviews. This feedback highlighted the need for robust handling of long-running tasks, seamless scheduling, and fault tolerance, leading to the introduction of Temporal.io into the architecture.
The inclusion of Temporal.io enables DIDImo to deliver enhanced capabilities, such as reliable state management, advanced scheduling, exponential retry mechanisms, and support for human-in-the-loop workflows. These benefits directly address user needs for scalability, resilience, and usability, making the platform more adaptable and effective in managing compliance workflows and reporting processes.
Through this iterative design process, DIDImo now leverages Temporal.io alongside a modular API and robust monitoring stack, ensuring it provides an intuitive, reliable, and user-focused conformance tool for verifying and managing compliance with European digital identity standards.
Step-by-Step Deployment
The deployment process for DIDImo is streamlined through a fully containerized infrastructure managed by Docker. Each component, from the backend services to monitoring tools, is packaged into containers, ensuring consistency across development, testing, and production environments.
- Containerized Services: All core components (e.g., API gateway, Temporal.io, PostgreSQL, Grafana, Prometheus) are deployed as Docker containers for seamless integration and scalability.
- Mock Services for Testing:
- Mailhog simulates mailing functionality, ensuring email workflows can be tested without external dependencies.
- A third-party simulator acts as a mock for external compliance tools, introducing intentional faults to stress-test the system.
- Additional external API mocks ensure all integrations are validated before real-world connections are established.
- Pre-Staging Validation: Before progressing to the staging environment, these mocks provide a controlled setup to test all workflows and API interactions comprehensively.
This containerized and mock-driven approach reduces dependency risks, accelerates testing, and ensures a reliable transition to staging and production environments.
Strategy Plan: From Development to Production
1. Development Phase
Containerized Architecture ✅:
- Ensure all services are fully containerized using Docker for consistent environments.
- Validate the configuration of core components such as Temporal.io, PostgreSQL, API gateway, and monitoring tools (Prometheus, Grafana).
Mock Services ✅:
- Deploy Mailhog for testing email workflows.
- Use the third-party simulator to test compliance integrations, including error scenarios.
- Set up external API mocks to replicate third-party dependencies for controlled testing.
Workflow Design 👷:
- Develop workflows using Temporal.io for compliance checks, reporting, and scheduling.
- Test workflows locally and validate state management, retries, and external signal handling.
- Integrate key components:
- Establish seamless connectivity between the Compliance Engine and external conformance tools via RESTful APIs.
Platform Interoperability 👷:
- Implement multi-standard support (eIDAS 2.0, EUDI Wallet, OpenID) to enable broad adoption across European ecosystems.
- Ensure backward compatibility with DIDroom's existing identity wallet infrastructure to extend TrustChain functionality.
2. Pre-Staging Testing
Integration Testing:
- Combine all services in a local environment with mocks to ensure end-to-end functionality.
- Validate all API endpoints, ensuring responses adhere to the Google JSON Style Guide.
User Experience Validation:
- Ensure the dashboard and UI are user-friendly, with clear navigation for developers and end-users.
- Gather feedback from stakeholders during this phase for iterative refinements.
3. Staging Environment
Isolated Deployment:
- Set up a staging environment mirroring production, with isolated database instances and Temporal namespaces.
- Replace mock services with real integrations where feasible.
Data Management:
- Load realistic test data into the staging environment to simulate production scenarios.
- Verify compliance workflows and reporting outputs with actual data.
Monitoring and Observability:
- Configure Grafana dashboards and Prometheus alerts to monitor system health and performance.
- Track workflow execution metrics via Temporal UI.
4. Pre-Production Validation
Security Audits:
- Perform a security audit of the system, focusing on API endpoints, data handling, and user access controls.
Stakeholder Approval:
- Present the staging environment to stakeholders and finalize any adjustments based on their feedback.
5. Production Rollout
Zero-Downtime Deployment:
- Use rolling updates to deploy the system into production, ensuring no downtime.
- Monitor system health during deployment to address any issues promptly.
Data Migration:
- If necessary, migrate existing data into the production database while ensuring integrity.
Post-Deployment Testing:
- Validate all workflows, APIs, and integrations in the live environment.
- Ensure Temporal workflows are functioning as expected with real-world usage.
6. Maintenance and Iteration
Continuous Monitoring:
- Leverage Prometheus and Grafana for ongoing monitoring and alerting.
- Use Temporal's debugging tools to inspect and resolve any workflow issues.
User Feedback Loop:
- Gather feedback from real users and stakeholders to refine features and improve the system.
Technical Components
The following components are critical to the DIDimo platform within the TrustChain ecosystem:
- Compliance Engine: Core module for conducting compliance checks and scoring.
- API Gateway: Facilitates secure access to all platform features and supports plugin submissions.
- Reporting Service: Generates comprehensive reports with detailed conformance scores.
- Dashboard and Comparison Tools: Provides a user-friendly interface for managing services and viewing public compliance data.
- Plugin Management System: Allows seamless integration of third-party conformance tools.
The Compliance Engine is composed in detail of the following services/containers
⚙️ didimo-backend (API Gateway)
didimo-didimo is the core service of the project, responsible for handling the primary application logic. It orchestrates all internal processes and ensures that the compliance engine and related functionalities operate cohesively. This is based on go, specially on pocketbase.io for the API/REST as a framework and Cobra for the CLI.
🔄 didimo-temporal
didimo-temporal is the Temporal Workflow service used for managing distributed workflows and task orchestration. It ensures reliability and scalability in handling complex operations by managing dependencies, retries, and task executions.
📧 didimo-mailhog
didimo-mailhog is a testing tool designed to intercept and inspect emails sent by the application. It allows developers to validate email functionalities during the development and testing phases without sending actual emails. This ensures that all email-related operations, such as notifications or verification links, work as intended.
RESOURCES
📊 didimo-grafana
didimo-grafana is a powerful visualization and monitoring tool integrated into the project to display metrics and logs from the system. It helps teams gain real-time insights into system performance, track key metrics, and debug issues through customizable dashboards.
🛠️ didimo-temporal-admin-tools
didimo-temporal-admin-tools provides administrative tools for managing the Temporal Workflow service. These tools allow developers to configure, monitor, and debug distributed workflows and ensure smooth task orchestration across the system.
👀 didimo-temporal-ui
didimo-temporal-ui is a user interface for managing and observing workflows in the Temporal service. It provides a visual representation of workflows, their statuses, and debugging tools, making workflow management more intuitive for administrators and developers.
📈 didimo-prometheus
didimo-prometheus is a metrics collection and monitoring tool integrated into the project. It scrapes, stores, and visualizes application metrics, enabling teams to track performance, detect anomalies, and optimize system behavior proactively.
🗄️ didimo-postgresql
didimo-postgresql is the PostgreSQL database instance used as the backend for data storage. It serves as the core storage layer for compliance data, user information, and reports, ensuring data integrity and efficient query handling.
🌐 didimo-thirdparty
didimo-thirdparty acts as a placeholder service for external integrations or third-party API interactions. This module facilitates seamless connectivity between the core system and various external services, ensuring interoperability and extensibility for future integrations.
RESOURCES
API Specification Evolution (Final)
Based on the user feedback we collected the following points
| Key Points | API Impact |
|---|---|
| Enhance plugin management systems to allow seamless integration of third-party tools. | Ensure plugin-related endpoints are modular and allow easy extensions. |
| Emphasize status transparency in long-running tasks (e.g., compliance checks). | Refine endpoints related to scheduling and task monitoring. |
| Improve response clarity for some endpoints. | Provide a clear distinction between error types and their resolutions. |
| Add sandbox environments for testing integrations. | Introduce sandbox APIs to test integrations. |
| Support batch operations for compliance checks. | Add batch processing capabilities for compliance and reporting. |
| Include multi-standard compliance in single API calls. | Enhance reporting APIs to support multiple standards per request. |
| Provide detailed conformance scores in compliance reports. | Refine the compliance score module to offer more granular insights. |
| Support team-based organization management. | Extend organization-related endpoints to include team management. |
| Enable feature-rich dashboard API for UI integrations. | Optimize dashboard endpoints to support complex UI widgets. |
| Include historical data in compliance results for trend analysis. | Update APIs to include historical compliance data. |
| Address privacy concerns and allow control over report visibility. | Add advanced privacy controls for reports and services. |
| Ensure interoperability with legacy systems. | Ensure all APIs are backward-compatible with legacy standards. |
| Add claim ownership APIs for organization management. | Extend endpoints to allow users to claim ownership of organizations. |
| Align error responses with the Google JSON Style Guide. | Revamp error handling to provide actionable recommendations in error messages. |
These can be summarized into the following:
Changes and Justifications
Based on feedback, the following changes were made:
- Added Plugin Management Endpoints: Supports third-party conformance tool integrations, addressing developer requests for extensibility.
- Enhanced Reporting APIs: Included granular compliance scoring and batch processing capabilities.
- Improved Privacy Controls: Introduced endpoints for managing report visibility, ensuring GDPR compliance.
- Introduced Batch Operations: Simplified compliance checks for multiple services at once, increasing efficiency.
- Error reporting: Ensure error handling aligns with the Google JSON Style Guide.
- Sandbox and testing environements: Add sandbox and testing environments of services like a fake/mock credential isuer or relying party to allow devs to test while developing their products.
Consistency
Core endpoints for compliance checks, organization management, and scheduling remain unchanged as they continue to meet project objectives and align with TrustChain’s vision. Also some endpoints like user management just changed the address to reflect directly the utilities provided by PocketBase.io that is the backend framework in go choosen that consists of an embedded database (SQLite) with realtime subscriptions, built-in auth management, convenient dashboard UI and simple REST-ish API.
Design Evolution
User feedback influenced iterative refinements:
- Functionality: Adds critical features like plugin management and batch operations.
- Usability: Ensures clearer error handling and privacy controls.
- Interoperability: Addresses legacy system support and multi-standard compliance. Added support for multi-standard conformance in single API calls, increasing system versatility.
- Scalability: Prepares the API for future growth by supporting advanced use cases.
Platform Identification and Integration
Selected Platform: DIDroom Integration
DIDroom serves as a foundational platform for DIDimo, providing an open-source identity solution that aligns with TrustChain’s goals of interoperability and decentralized identity management.
Evaluation
- Interoperability: Supports integration with DIDroom’s existing W3C-compliant identity wallets, facilitating cross-standard compatibility.
- Scalability: DIDroom’s modular architecture ensures the ability to handle growing transaction volumes and new compliance demands.
- Security: Integrates robust cryptographic protocols to maintain data integrity and secure operations.
- Regulatory Compliance: Fully adheres to GDPR and European standards for privacy and governance.
- Customization: Allows the extension of existing DIDroom libraries to support TrustChain-specific use cases.
- Efficiency: Optimized workflows reduce redundancy, leveraging DIDroom’s reusable identity management components.
Integration with Existing Tools
DIDimo leverages components and tools from DIDroom to strengthen the TrustChain ecosystem:
- Identity Wallets: DIDroom identity wallets provide a secure basis for credential management.
- Libraries: Extends decentralized identity libraries already developed under DIDroom.
- External Tools: Integrates with external conformance and testing tools with the power of our workflow management.
For now we have planned to use as external conformance platforms, some based on GITB that is choosen by the european commision as the standard testbed platform for all different projects. In particular:
- EWC wallet conformance tool https://github.com/EWC-consortium/ewc-wallet-conformance-backend
- OpenID Conformance suite to validate the openid4vp and in general the openid4vci standard https://gitlab.com/openid/conformance-suite/
- The test tools suggested by the Wallet Interoperability Special Interest Group (WISIG) of the OpenWallet Foundation https://tac.openwallet.foundation/SIGs/wallet-interoperability/
- The EBSI conformance testing platform https://hub.ebsi.eu/wallet-conformance