Senior Data Engineer

About the job

Job Purpose: We are looking for a highly skilled Senior Data Engineer with a strong background in Databricks, Azure Data Services, and modern data engineering tools to design, build, and optimize scalable data pipelines. The ideal candidate will have at least 5 years of hands-on experience with Databricks and 5+ years of experience in data engineering, with proven expertise in ETL/ELT development, Spark-based processing, and Python/SQL programming. Key Responsibilities: • Design, build, and manage robust, scalable data pipelines using Databricks, Delta Live Tables, and Azure Data Factory (ADF) or Fivetran. • Implement advanced ETL/ELT processes to ingest and transform data from multiple sources, including SAP. • Develop and maintain reusable code using Python, SQL, and PySpark. • Work with Unity Catalog to manage data governance, access controls, and data lineage across environments. • Use Databricks notebooks and Git for collaborative development, version control, and CI/CD. • Leverage Azure services and Terraform for provisioning, automation, and cloud infrastructure management. • Optimize data models for performance, scalability, and operational efficiency in a cloud-native architecture. • Collaborate closely with Data Analysts, Scientists, and Business Stakeholders to deliver clean, trusted datasets. • Document technical solutions, data flows, and transformation logic. Requirements: Qualifications: • Bachelor's degree in Computer Science, Engineering, Information Systems, or related field. • Databricks certification (e.g., Databricks Certified Data Engineer Associate/Professional) preferred. Experience: • 5+ years in data engineering or big data development roles. • 5+ years hands-on experience with Databricks, including DLT and Unity Catalog. • Experience working with Azure Data Factory (ADF) or Fivetran for ETL/ELT. • Strong programming skills in Python and SQL. • Proficiency in PySpark or Apache Spark for distributed data processing. • Experience with Git, notebook-based development, and CI/CD pipelines. • Familiarity with Azure administration and infrastructure as code (Terraform). • Exposure to SAP systems as a data source is a plus. Soft Skills: • Strong problem-solving skills and attention to detail. • Ability to work collaboratively in cross-functional teams. • Excellent communication and documentation skills. • Comfortable working in Agile environments.

Applicants must have on their profile

• Skills
• Qualifications
• Certifications
• Work experience
• Right to work in South Africa