Image storage system with deduplication
Reducing disk space costs through an efficient algorithm for detecting pixel-level duplicate images
Tasks
- Develop an image deduplication system at the pixel data level
- Implement a web service for image storage and processing
- Integrate into the CI/CD pipeline for report generation
- Automatically generate previews and optimize images for display in web reports
- Implement computer vision-based algorithms for comparing with reference images with adjustable sensitivity thresholds
About the Project
Testing 3D rendering engines generates a large number of images, many of which are pixel-identical across different product versions. This led to excessive disk space consumption and increased infrastructure costs.
We developed an image storage system with deduplication, which significantly reduced the volume of stored data and accelerated verification processes.
Results
Challenges and Solutions
- Image Deduplication
- To improve accuracy, we implemented pixel-level image comparison with a configurable difference threshold. This eliminated false duplicates and preserved only truly unique images.
- Integration into Reports
- The system automatically generates image links in reports, creates previews, and optimizes image sizes for optimal browser display.
- Performance and Scalability
- The system processes tens of thousands of images monthly without slowing down CI/CD processes. Upload queues, caching, and image transformation optimization were implemented.
Details
To address excessive resource consumption in storing reports and 3D rendering test results, an image storage system with deduplication was developed. We moved away from traditional file-level approaches, as identical images could have different encodings.
A system was implemented to compare images at the pixel data level and compute hashes resilient to encoding differences. This achieved high accuracy in detecting duplicates while maintaining performance during new data uploads.
The system’s web interface allows convenient image viewing, comparison with reference images, and tracking changes across product versions.
Related Services
Outsourcing
Our team of skilled developers is ready to create websites and applications of any complexity. We build modern web solutions using cutting-edge technologies and architectural approaches.
Read MoreReady to discuss your project?
Describe your task, we will make a research and respond to you as soon as possible.