Consultant | Civil Government Client
Summary
As a junior data scientist on a high-stakes government security project, I spearheaded the automation of a critical defensive monitoring tool designed to protect a data warehouse. My focus was on transforming manual Python scripts into robust, scalable ETL pipelines using bash and Python, ensuring the system operated reliably 24/7. This involved building four pipelines to automate daily data refreshes from five+ sources, while implementing error handling, logging, and retry mechanisms to prevent critical failures.
The challenge extended beyond code: the client’s legacy systems required creative workarounds, including robotic process automation (RPA) via UiPath to extract data from proprietary repositories and distribute reports. Despite limited RPA experience, I led a team through certification, debugging, and navigating complex approval processes to deploy three attended automations in a month—cutting manual work by 40 hours weekly.
This work underscored the importance of pragmatism, cross-functional collaboration, and building systems that reduce complexity while safeguarding sensitive data. Automation wasn’t just about efficiency—it was about reliability, trust, and empowering experts to focus on what truly matters.
Impact
- Led the development of 7 automation solutions using Python, cron jobs, and RPA (UiPath), ensuring 24/7 reliability of a critical defensive monitoring tool for a government agency.
- Managed a team of 2 in RPA development, overcoming legacy system challenges to deploy 3 attended automations that reduced manual entry by 40 hours per week.
- Designed scalable ETL pipelines and error-handling frameworks, enabling seamless data integration across 5+ sources and reducing operational risk in a high-stakes security environment.
Tech Stack
- python (psycopg2, sqlalchemy, pandas, numpy, subprocess, argparse)
- BASH (CRON)
- RPA (UiPath, PowerAutomate, Windows Task Scheduler)
- SQL (Postgresql)
- Tableau
Read the full story here on my Substack.