TechOps Examples
Hey — It's Govardhana MK 👋
Welcome to another technical edition.
Every Tuesday – You’ll receive a free edition with a byte-size use case, remote job opportunities, top news, tools, and articles.
Every Thursday and Saturday – You’ll receive a special edition with a deep dive use case, remote job opportunities and articles.
Give Cursor, Claude, and Copilot the organizational knowledge they need to generate code that reflects how your system actually works.
Unblocked synthesizes your code, PR history, conversations, docs, and runtime signals to surface the right context for each task.
Generate plans, code, and reviews that reflect how your system actually works
Reduce costly retrieval loops and tool calls by providing better context up front
Spend less time correcting outputs for code that should have been right in the first place.
“This is the best application of AI’s potential in enhancing developer proficiency.” - SVP Engineering at AppDirect
IN TODAY'S EDITION
🧠 Use Case
How to Design a Scalable File Upload Architecture in AWS
👀 Remote Jobs
Flashbots is hiring a Senior DevOps Engineer
Remote Location: Worldwide
Metabase is hiring a Senior SRE/DevOps Engineer
Remote Location: Worldwide
Powered by: Jobsurface.com
📚 Resources
If you’re not a subscriber, here’s what you missed last week.
To receive all the full articles and support TechOps Examples, consider subscribing:
One-time 25% OFF on all annual plans of memberships. Closes Soon.
🧠 USE CASE
How to Design a Scalable File Upload Architecture in AWS
Imagine you're developing a document management mobile app, where users can upload large files such as PDFs, presentations, and spreadsheets. The app needs to store these documents securely and efficiently.
For this, utilizing AWS services like API Gateway and S3 is an ideal choice due to their scalability, security, and ease of integration. A mobile client can directly upload files to an S3 bucket via API Gateway, as depicted below:
While this might seem straightforward, it has a critical limitation: API Gateway imposes a payload size limit of 10MB.
Mobile Client sends a request to API Gateway.
API Gateway attempts to upload the file to S3 Bucket.
Files under 10MB succeed, while those over 10MB fail.
This architecture is not only inefficient but also unsuitable for applications dealing with larger files.
Key considerations for API Gateway:
Feature | Value/Range |
|---|---|
Maximum routes per API | 300 |
Maximum integrations per API | 300 |
Maximum stages per API | 10 |
Maximum request payload size | 10 MB |
Maximum response payload size | 10 MB |
Request rate limit | 10,000 requests per second |
Burst limit | 5,000 requests per second |
Have you heard about Pre-Signed URLs?
A pre-signed URL is a URL that you can provide to your users to grant temporary access to a specific S3 bucket. With this URL, users can upload files directly to S3 without requiring AWS credentials or going through an intermediary service like API Gateway.
Pre-signed URLs are time-limited and can be configured to allow only specific actions (like uploads), ensuring secure and controlled access to your S3 buckets. Let's transition to a better architecture as below:
Mobile Client sends a request to API Gateway.
API Gateway invokes a Lambda Function.
Lambda Function generates a pre-signed URL for the S3 bucket.
Lambda Function returns the pre-signed URL to API Gateway.
API Gateway forwards the pre-signed URL to the Mobile Client.
Mobile Client uses the pre-signed URL to upload the file directly to S3 Bucket.
Why is this the Better Choice?
No File Size Limitations: By allowing direct uploads to S3, the file size constraints of API Gateway are eliminated.
Enhanced Performance: Direct uploads reduce latency and improve the overall performance of the file upload process.
Scalability: This architecture can handle a larger number of uploads without additional complexity.
Security: Pre-signed URLs ensure that only authorized users can upload files, maintaining security without sacrificing convenience.
This way, you ensure a seamless, scalable, and efficient file upload process for your mobile app, enhancing user experience and operational efficiency.
In my experience, what I understand is 'architecture' design is a subjective choice and differs drastically with individuals and systems. This solution is one of the choices.
Unblocked synthesizes your code, PR history, conversations, docs, and runtime signals to surface the right context for each task.
🔴 Get my DevOps & Kubernetes ebooks! (free for Premium Club and Personal Tier newsletter subscribers)
Looking to promote your company, product, service, or event to 54,000+ DevOps and Cloud Professionals? Let's work together.
