Data Engineering, Team Lead

Job description

Responsibilities :

As the Data Engineer Team Lead, you will play a crucial role in overseeing the design, development, and optimization of our data pipelines, ensuring seamless integration and availability of data across all systems. You will lead a talented team of data engineers and spearhead the execution of data strategies that are integral to supporting critical business decisions and operations. A significant responsibility is to develop a unified data pipeline that effectively serves both internal systems and external applications, ensuring secure and real-time data flow across all platforms.

Team Leadership: Lead and mentor a team of data engineers to design, build, and optimize scalable data pipelines for analytics, reporting, and operational needs.

Unified Data Pipeline Management: Create and manage a unified data pipeline that integrates multiple data sources, facilitating efficient access for both internal and external systems, including third-party tools and client-facing platforms.

Utilization of Low-Code/No-Code Tools: Leverage low-code/no-code integration tools such as Fivetran or RudderStack to accelerate data workflows and improve integration efficiency.

Cross-Functional Collaboration: Work closely with various teams to identify data needs and ensure smooth data flows between systems, enhancing high availability and integration.

Data Governance and Quality Assurance: Implement and manage data governance and quality standards, ensuring data consistency and reliability throughout the organization.

Real-Time and Batch Data Architecture: Architect and oversee the implementation of real-time and batch data pipelines using data streaming tools like Kafka or Pub/Sub, ensuring low-latency, high-performance data processing.

Technical Guidance and Support: Provide technical guidance to your team, aiding in troubleshooting complex data issues and optimizing performance and cost-effectiveness while adhering to best practices.

Continuous Improvement: Stay informed on the latest in data engineering technologies and best practices, especially concerning real-time processing and low-code/no-code integrations, to enhance our data infrastructure continually.

Stakeholder Coordination: Communicate with stakeholders to ensure timely delivery of data solutions that effectively meet business requirements.

Requirements :

Education: Bachelor's degree in Computer Science, Data Engineering, or a related field.

Experience: 10+ years in data engineering, with at least 3+ years in a management or leadership role.

Expertise in designing, building, and maintaining data pipelines and data warehouses.

Technical Skills :

Strong knowledge of cloud platforms (Google Cloud or AWS) and familiarity with tools like BigQuery, PostgreSQL, and Airflow.

Proficient in programming languages such as Python, with experience in ETL/ELT processes.

Hands-on experience with low-code/no-code data integration tools.

Deep understanding of real-time data processing and streaming technologies such as Kafka or Pub/Sub.

Understanding of Data Frameworks: Experience in building unified data pipelines for secure and consistent data flows, working with both real-time and batch processing systems.

Governance Knowledge: Strong understanding of data governance, security, and compliance best practices.

Communication Skills: Excellent communication abilities with a talent for explaining complex technical concepts to non-technical audiences and mentoring junior team members.

Job Type: Permanent

Pay: RM15,000.00 - RM20,000.00 per month

Benefits:

• Free parking
• Health insurance
• Maternity leave
• Opportunities for promotion
• Parental leave
• Professional development
• Work from home

Experience:

• leadership/management: 3 years (Required)
• data engineering: 10 years (Required)