Tech Audit Q&A
From the details of our source code repository and architecture overview to database schema documentation, API specifications, deployment and configuration guides, and dependency documentation, delve into the robust foundation that powers our innovative financial technology solutions.
Source code repository
Details of the version control system (e.g., Git, SVN) where the source code is stored. Currently, we are using GIT with GitLab. For source code transfer and updates, we provide two ways:
- Pull, your repository sync from our side
- Push, we deliver updates directly to your repository
Architecture overview
Architectural overview document that describes the high-level design of the system, including components, modules, and their interactions.
High-level architecture
Application components
Deployment model
Technical architecture
Schematic diagram of the environment
Template for disaster recovery / fault-tolerant architecture
Parameters for minimal configuration:
Front end
- CPU – 1
- RAM – 2 GB
- SSD – 40 GB
- OS: Ubuntu 22.04 LTS
- Software: NGINX 1.18.0
Application server
- CPU – 4
- RAM – 8 GB
- SSD – 160 GB
- OS: Ubuntu 22.04 LTS
- Software: openJDK 17
Postgres DB
- CPU – 4
- RAM – 16 GB
- SSD – 160 GB
- OS: Ubuntu 22.04 LTS
- DB: PostgreSQL – 14.4
Transaction Viewer DB
- CPU – 2
- RAM – 4 GB
- SSD – 80 GB
- OS: Ubuntu 22.04 LTS
- DB: MongoDB – 6.02
Database schema documentation
Documentation that outlines the structure of the database schema, including tables, columns, relationships, and indexes.
We are using a Code-driven DB approach. All system activity is mapped automatically in DB via Hibernate.
API documentation
We are using an API-First approach and all features are available via APIs. We are using the SWAGGER (Open API 3.0.3) format. The latest version is always available here, updated daily: SDK.finance back-office demo.
Testing documentation
Documentation related to testing, including test plans, test cases, and test results. This is to ensure the reliability and quality of the codebase.
Currently, we have:
- Collections in Postman, if you want to review it let us know and we clean it up from internal passwords and share it. It takes some time because it is pretty big.
- We are migrating test cases writing in Test Rail is in progress (435 currently available, QAs are working to update and expand it, it will be about double more after we finish)
- Unit tests in code (if relevant for this point). Application functionality is covered by numerous unit and integration tests.
- For unit and integration testing, we use:
Monitoring and logging documentation
Documentation on how the application is monitored and logged, including tools used, log formats, and monitoring/alerting thresholds. We are using an integrated Java Spring engine Actuator for low-level monitoring. Additionally, we recommended that our clients use a more advanced framework for overall system performance, health check monitoring, and alerting like:
Or available from Infrastructure providers, for example:
AWS
MS Azure
Security documentation
Documentation related to security measures implemented in the application, including authentication mechanisms, authorization policies, data encryption, and vulnerability assessments.
Secure Software Development Standard:
- Auth tools: Native JWT, OAUTH 2 Native or SSO via KeyCloack
- Application encryption: SHA 2.1, HmacSHA256
- Code quality control: LLM’s with IDE (Amazon Code Whisper(Q) and Git Co-pilot)and Manual Code review
- Static code analysis: SonarQube or JetBrains Qodana.
Check SDK.finance system performance testing to explore the Platform workload capacity.
System architecture and resilience
Impacts of failure
The pre-production and production environments are isolated, so a failure in one does not directly impact the other. This setup allows for testing and bug-fixing stages without affecting the live environment.
Application server side
The system is stateless, and all activity is mapped on an ACID type database. Both MongoDB and PostgreSQL used in SDK.finance are ACID compliant. If an application server goes down, the state of the system is always available for another application server copy. The application server and DB copy must be supported with an Active-Active type of “Disaster Recovery” infrastructure, where the DB and application servers are continuously synchronized in real-time from the DB replica node. With high traffic, the application server can operate as a cluster of several application server instances behind a load balancer. In case one of the cluster nodes goes down, the remaining traffic is distributed to the other cluster nodes.
Database management
Each user’s database is isolated and managed by the user. This isolation means that a failure in one user’s database does not impact others. DB management is also a part of the daily duties of the DBA position in the company and relies on the available toolset. Some tools are available only for clouds, while others can be deployed in on-premises data centers. Continuous monitoring, backups, and data-retention policies are daily duties of the IT support team.
API gateway
The SDK.finance platform uses an API gateway to manage requests. If one service fails, the gateway can route traffic to fallback services or another node in the cluster. In the SDK.finance system, we have implemented an exception-handling mechanism designed to ensure continuity of operations in the event of an error. This mechanism isolates errors that occur in specific processes or during the execution of individual operations. By effectively catching and managing exceptions locally, the system prevents these errors from impacting the stability of the entire system. This allows other operations to continue functioning normally, thereby enhancing the overall resilience and reliability of our application. Additionally, the API orchestration layer can be extended with third-party tools that operate as a high-level API management tool plus an additional security layer. We recommend using KONG as a tool we have successfully used in production.
Single points of failure and mitigation
Database server
If the database server fails, it can bring down the entire application. As mentioned before, DBs must continuously have a backup stage for long-term storage, plus the Active-Active Disaster Recovery stage that intercepts transactions almost in real-time if the main DB goes down for some reason. Internal mitigation with the IT support team must include setting up database replication, backups, and failover mechanisms.
API gateway
If the API gateway fails, all services dependent on it will be impacted. Using multiple instances of the gateway and load balancing can mitigate this risk, as mentioned earlier, or with a switch to the Disaster Recovery stage.
Application server
A failure in the application server can halt all processing. Running multiple instances and using load balancers can distribute traffic and avoid a single point of failure, or with a switch to the Disaster Recovery stage.
Code-first approach benefits
How our code-first approach enhances performance, source code maintenance, and adding new modules?
We use a Code-First approach to the database structure, and here are its advantages:
Enhancing performance
Using the Code-First approach, developers can optimize the database structure based on the application’s needs rather than being confined to a preset schema. This allows for more efficient use of database resources, such as creating indexes and optimizing queries based on the actual data usage by the application.
Source code maintenance
The Code-First approach facilitates source code maintenance as changes to the data model can be implemented directly in the code, without switching between different tools or languages (like SQL). All changes are centralized in the code, simplifying version control and change management.
Adding new modules
Expanding the application with new functional modules becomes easier, as new entities or changes to existing entities can be seamlessly integrated into the overall database structure. Code-First allows for automatic database migrations, significantly speeding up the development process and simplifying the integration of new components. It also significantly reduces the time for DB management and CI/CD pipelines for updates.
Overall, the code-first approach offers flexibility and manageability in development, greatly simplifying the processes of scaling and adapting the system to changing business requirements.
Adding, removing, and scaling modules
Removing or adding new modules
Each module has its own DB migration scripts, but tables are not independent of each other. Some fields have references (PK, FK) to tables that have mapped entities in other modules. To decouple, these relationships should be removed (additional migrations and changes in the codebase), but it’s doable after further discussion.
Scaling up a single module of the application to accommodate increased user traffic or data growth for that module.
The module is a part of the project until it is refactored to a stand-alone service. If we need more copies of the module or service, we just run more instances, and the load balancer sends traffic of certain API calls to a cluster of instances. It’s easy to do with the KONG API orchestration solution we mentioned above.
Can the current project easily be integrated with other microservices with their data storage?
Integration with microservices: The SDK.finance platform is designed with flexibility in mind, allowing for easy integration with other microservices. Each service can maintain its data storage, ensuring seamless and efficient integration. Other microservices can easily communicate with SDK.finance modules or services by using an Enterprise Message Bus such as KAFKA which SDK.finance supports out of the box. Alternatively, it can be integrated with similar services like RabbitMQ and others.
Complexity
Codebase complexity: The modular design of the codebase ensures that it remains manageable as the application grows. Each module is self-contained, making it easier to understand and maintain. This approach reduces the complexity of the overall system.
Monitor the health and performance of various parts of the application individually
In the Java Spring Framework, there is an Actuator module available for use in the project. This tool helps monitor and manage the application in production at the lowest tech layer with multiple metrics (https://www.baeldung.com/spring-boot-actuators). Additionally, business-related monitors, log representation, and alerting can be done with multiple available tools. We recommend NewRelic if it’s allowed, and the ELK stack for on-premises data centers with a free edition.
The leniency of the selection of the database engines with alternative DBMSs (MySQL or Cassandra..etc)
In the application, Spring Data JPA allows implementing communications between applications and different types of DBMSs. We also use Hibernate, which minimizes the impact on developers and abstracts data manipulation with any of the popular DBs of communication dialects.