Sammy

Associate Data Engineer

Shell • Full time

Aug 2024 - Present • 1 yr 11 mos

• Owned end-to-end SLA-critical batch data pipelines using PySpark and Spark SQL on Azure Databricks, supporting financial reporting, regulatory submissions, and downstream analytics. • Designed and implemented incremental and idempotent processing frameworks to ensure data completeness, consistency, and safe reprocessing, reducing operational risk in regulatory and financial datasets. • Optimized Spark SQL workloads through effective partitioning, join strategies, and query tuning, improving batch performance and reducing cloud compute costs, resulting in of approximately $170K year-on-year savings. • Orchestrated enterprise-grade batch workflows using Control-M, implementing dependency management, automated retries, monitoring, and alerting to ensure consistent SLA adherence in 24×7 production environments. • Led platform modernization initiatives, migrating 500+ production batch jobs with zero downtime while maintaining 100% SLA compliance. • Developed automated data validation, reconciliation, and audit control frameworks, improving regulatory compliance and enabling early detection of data quality and completeness issues in production. • Designed centralized metadata management and RBAC frameworks, standardizing dataset access controls and improving developer and analyst onboarding efficiency by approximately 40%. • Built reusable backend frameworks and reference architectures, saving 120+ engineering hours across multiple delivery teams. • Partnered with business and leadership stakeholders to deliver Power BI dashboards for fuel and EV pricing analysis, enabling data-driven decision-making and executive reporting.