Success Case Study #2 - Automate backdated image retrieval & storage

Following up on my previous battles with manual operational flows, I discovered another hidden technical debt.

I realized that historical manual image update processes had only been pushing the updated imagesto our partner’s system. We had completely forgot to store these crucial files inside our own internal system! That is absurd!

Problem Space

Job Story

• If we needed to audit or verify a past image update, we had a major data gap.
• To meet compliance requirement and complete customer profiles, we needed to pull these historical images back into our cloud storage and log them correctly in our internal database with exact backdated timestamps.

High-level problem mapping

To resolve this issue, the process for every single user_id looked like this.

Doing this manual retrieval for a thousand customers was incredibly painful. It was a ridiculous task which consumed more than a week of my bandwidth. Please, I could not just copy and paste more than a thousand files.

Solution Space

Ideation

As a Product Manager, I could not spend a whole week acting as copy-paste guy. And I already did an incredible job for previous task, so it's not impossible to automate this stuff.

Final Solution

Without brute-force by manual process, I collaborated with AI to write a Python automation script.

Here is how the final flow operated:

1. Data Preparation: Created a simple user_data mapping file containing our internal user_id alongside the partner's external_user_id.
2. SFTP Automation:
   1. Setup an SFTP connection from Python script. The script automatically scanned the user_data file.
   2. For each external_user_id, the script accessed their respective SFTP folder, retrieved the specific images in format.
   3. Finally, stored them into a cleanly formatted user_id folder in our cloud storage.
3. Data Mapping: Retrieved the historical update ID records (through manual process) to map the exact backdated data to each corresponding user_id.
4. Final Deliverable: The end result was a combined package: a history record of past updates with accurate backdated data, paired with the corresponding image folders for each user_id.

Impact

• Massive Time Savings: What would have taken more than a week of manual workload for a thousand customers was reduced to just a single day of script execution and validation.
• Streamlined Engineering Handoff: Provided the engineering team with perfectly structured data and folders, they just needed to upload them to internal system.

Reflection & Lessons Learned for Future PMs

As Product Managers, our time is one of our most valuable resources. When you encounter a manual task, just pause a little bit.

• Leverage AI as your assistant: You don't need to be a software engineer to write functional scripts. Prepare logic and communicate to AI tools can turn a nightmare into a few hours of automation work.
• Look for systemic gaps: The root cause here was a gap in the original process design (updating the partner but not internal storage). Always ensure data flows are holistic across both internal and external ecosystems.
• Automate the boring stuff: Your primary job is to solve user problems and drive business value, and if a machine can do it, let them do it.
• Keep a cold head: Mistakes can always be happened, but always be patient enough to solve it.