m
mustafainayat

mustafainayat

Interactive, Collaborative, Robust Personality holder

Pakistan
Inglese
Alcune informazioni sono riportate in lingua inglese.
Chi sono
I’m a Data Engineer with 5+ years of experience building and operating production-grade ETL/ELT pipelines. I work with Python and SQL to ingest data from APIs, files, and databases, orchestrate workflows using Apache Airflow/Win TS, and implement scalable transformations and models using dbt. I also have strong experience with Talend and relational databases including MySQL, PostgreSQL, and MS SQL Server, with a focus on data quality, performance tuning, monitoring, and maintainable pipeline design.... Continua a leggere

Competenze

m
mustafainayat
mustafainayat
offline • 

Consulta i miei servizi

Dati ETL
I will design and build production ready etl pipelines and data transformations
A partire da75 USD / progettoMaggiori dettagli

Esperienza lavorativa

Sr. Data Engineer

Afiniti • Full time

Feb 2021 - Present • 5 yrs 3 mos

Designed, developed, & maintained data engineering solutions to automate daily ETL/ELT workflows, ensuring accurate & timely preparation of datasets for downstream modeling & analytics. Built robust pipelines in Python, integrating with PostgreSQL, MySQL, MSSQL & leveraging dbt for transformation logic. Implemented API-driven communication between distributed system components to ensure seamless integration. Applied best practices in version control, continuous integration, & deployment using Bitbucket & automated CI/CD pipelines. Utilized Visual Studio Code for development, & managed project deliverables & progress tracking through Jira. Collaborated with cross-functional teams to architect scalable solutions, optimize performance, & maintain code standards. Orchestrated the deployment on Afiniti's premier client, consistently yielding monthly revenues exceeding €5 million. Championed the development & management of a comprehensive ETL pipeline, seamlessly accommodating approximately 60 million records daily, while overseeing end-to-end operations, including ETL, ELT, modeling data, & data loading pipelines. Pioneered the development of Python-based utilities to facilitate data transfer across diverse databases, leveraging PySpark, PXF Utility, & Dataframe (DF) Transfers. Innovated Python tools to automate the generation of model checkpoints, data validation, enhancing overall efficiency. Engineered a tailored Error Management System (EMS) into the pipeline, enhancing data integrity & reliability. Leveraged PowerBI to craft analytical reports, forecasting revenue trends, & furnishing other valuable insights. Pioneered the development of a Customer Scoring utility utilizing R-Studio, enhancing data-driven processes