Lead Data Engineer

RESPONSIBILITIES:

As a Data Engineer, you will be responsible for designing, developing, and maintaining data solutions for data generation, collection, and processing.

Provide technical guidance and mentorship to the data engineering team.

Set technical standards and best practices for Data Lake & AI platform projects.

Design and architect scalable, reliable, and secure data pipelines and infrastructure.

Oversee the development and maintenance of ETL (Extract, Transform, Load) processes.

Integrate data from various sources, ensuring data consistency, quality, and reliability.

Work on data modelling and schema design to support business requirements.

Optimize data processing jobs for performance and scalability.

Monitor and improve the performance of data storage solutions.

Coordinate with data scientists, analysts, and business stakeholders to understand data requirements.

Facilitate communication between technical and non-technical teams.

Plan and manage data engineering projects, including develop and maintain comprehensive documentation for data processes, schemas, and data dictionaries. task assignment and progress tracking.

Actively manage cloud-based data services to balance performance and cost.

Ensure projects are delivered on time and within budget.

Collaborate with cross-functional teams to identify and resolve data-related issues.

Stay current with the latest data engineering technologies, tools, and best practices.

DESIRABLE SKILLS & QUALIFICATION:

Capable of developing data platforms.

Proficiency inETL tool Talend, in addition other ETL knowledge is added advantage.

Strong proficiency with Python/ SQL and NoSQL databases (Hbase) and BI tools (Power BI) for data analysis and report generation.

Knowledge of distributed computing frameworks such as Hadoop, Spark, Hive, Presto etc.

Expertise in cloud data warehousing and core data modelling concepts.

Proficiency in version control systems (Git) and experience with CI/CD pipelines.

Deep familiarity with a cloud-native, modern data engineering technology stack compatible with Alibaba and AWS.

MUST have proven experience in leading and/ or managing a team of Data Engineers previously.

Confident communicator, capable of advising and influencing senior stakeholders.

Certifications: Degree in Computer science, Data Engineering Certification is a plus.

Experience : 10 Yrs Plus.