Domatic is a technology and service company that is deeply invested in data aggregation, analytics, and automation. With years of expertise, Domatic provides high-quality, scalable, and customizable services across industries where large-scale data collection is key, helping businesses solve complex problems with data and technology.
The client required a robust backend architecture to scrape data from specific websites, store the status of the scraped data, and enable CSV downloads of the data. By leveraging AWS Serverless services and AWS SAM, they efficiently deployed the necessary infrastructure and applications through AWS CodePipeline, ensuring scalability, cost-effectiveness, and rapid development to meet the client’s needs.
This case study highlights their successful multi-environment deployment of a backend architecture for a web scraping service, enhancing its data scraping capabilities.
Key Challenges: Complex Requirements for a Scalable Backend
Domatic faced several challenges in developing a reliable backend architecture that met their needs. These challenges included:
- Data Scraping Efficiency: The backend architecture needed to scrape data from specified websites reliably and efficiently.
- Data Management: Storing the status of the scraped data in a database required a robust and scalable solution
- CSV Download Functionality: The system needed to offer a feature to download the scraped data in CSV format, ensuring ease of access and usability
- Multi-Environment Deployment: The architecture had to support deployment across multiple environments (development, testing, and production) without compromising scalability or cost-effectiveness
- Rapid Development and Deployment: The client required a solution that enabled quick development cycles and deployment, allowing for rapid iteration and adaptation
- Cost Efficiency: Ensuring that the solution was cost-effective, especially in scaling resources up and down based on demand, was crucial to the client’s budget
TekBay’s Solution: Tailored for Scalable and Cost-Effective Deployment
Recognizing the need for rapid deployment and flexibility, TekBay determined that a serverless architecture would be the optimal solution for them. The serverless approach allowed the team to quickly set up the project, delivering immediate results and handling various scenarios with ease.
Key components of the implementation included:
Serverless Services: These services allowed for effective management of service limits, avoiding default error pages, and preventing application crashes under high traffic conditions. The pay-per-use billing model further enhanced cost efficiency by scaling down resources during periods of no load.
AWS SAM and CodePipeline: The entire stack was defined in a SAM template, an extension to CloudFormation, and deployed across multiple environments (development, testing, and production) using CodePipeline.
This SAM template enables simple deployment combined with parameter_overrides for seamless management of multiple application environments. Cross-account IAM roles ensured consistent deployments across environments through a multi-stack solution.
CodeBuild: During the build stage, CodeBuild was used to test and build the handlers, executing the aws cloudformation package command to generate the packaged template.
aws cloudformation package --template-file stack-template.yaml --s3-bucket <s3-bucket>
--output-template-file packaged-template.yamlThis template was then passed to subsequent stages, where it was deployed to each environment using CloudFormation with the StageName parameter in parameter-overrides
aws cloudformation deploy --template-file packaged-template.yaml --stack-name <StackDev|StackStaging|StackProd>
--capabilities CAPABILITY_NAMED_IAM --parameter-overrides StageName=<Dev|Staging|Prod>
Monitoring and Rollback: Essential metrics were tracked using AWS CloudWatch, and canary deployment techniques were utilized to manage production rollbacks. The CI system, powered by CodePipeline, automated the entire process, from commits and tests to handler compilation and deployment to development stacks.
Benefits
The use of AWS Serverless services and AWS SAM allowed for rapid development and deployment, ensuring that the client could quickly adapt to changing requirements while maintaining a robust backend infrastructure. This approach also provided significant cost savings and reduced the complexity of the development process.
Key benefits of the solution included:
- Fast and Agile Development: The serverless architecture facilitated rapid development and deployment, reducing overhead and enabling quick iteration and adaptation to client needs.
- Cost Efficiency: The serverless environment provided a low-cost platform for development, testing, and experimentation, with a potential 70-90% reduction in server costs due to the pay-per-use model.
- Flexible Development Environment: The solution supported multiple programming languages, offering a versatile development environment.
- Reduced Complexity: The compositional capabilities of serverless services significantly reduced the amount of coding required and avoided challenges associated with running applications on resource-constrained machines.
Results
The serverless architecture implemented by TekBay for Domatic not only met but exceeded the client’s expectations by providing a scalable, cost-effective, and agile solution.
- Infrastructure Code Coverage: 90% of infrastructure is managed with AWS SAM and CloudFormation, ensuring consistency and automation.
- Configuration Drift Rate: 2% drift, controlled through automated detection and correction.
- Time to Provision Infrastructure: Average provisioning time is 8 minutes per component using AWS CodePipeline.
- Mean Time to Recover (MTTR): 1 hour recovery time using automated rollbacks with AWS CodeDeploy.
- Deployment Frequency: Multiple daily deployments with minimal manual intervention.
- Cost Optimization: 40% cost reduction by scaling serverless architecture dynamically.
In conclusion, Domatic achieved scalability, reliability, and cost savings for their web scraping service through the successful utilization of AWS Serverless services and AWS SAM. The multi-environment deployment strategy, implemented via AWS CodePipeline, enabled them to efficiently build and maintain their scraping application and the supporting infrastructure, ultimately ensuring the project’s success.
If your organization is looking to achieve similar results with scalable, cost-effective, and rapidly deployable solutions, consider partnering with TekBay Digital Solutions. Contact us today to learn how we can help you transform your backend architecture with our expert serverless services.
