Job Description
We are looking for a highly experienced and motivated Azure Data Engineer to join our team. The successful candidate will have a minimum of 5 years of hands-on experience in Azure and Databricks architecture, migration, and development. This role requires excellent communication skills and the ability to own and self-manage work. The candidate must be able to overlap with US CST work hours for up to 6 hours daily.
Key Responsibilities:
- Design, develop, and implement advanced analytics solutions using Azure and Databricks.
- Perform data migration and transformation tasks on Databricks.
- Develop and maintain scalable data pipelines and build out new API integrations to support continuing increases in data volume and complexity.
- Work closely with data engineers, data scientists, and business stakeholders to understand their requirements and provide expert guidance.
- Optimize and tune the performance of Databricks and Azure-based solutions.
- Manage and monitor the Databricks environment to ensure reliability and performance.
- Lead the architecture and development of complex data integration projects.
- Conduct code reviews, provide feedback, and ensure best practices are followed.
- Own and self-manage assigned tasks, ensuring timely delivery with high quality.
- Collaborate with cross-functional teams and stakeholders in a remote work environment.
Qualifications:
- Bachelor’s or Master’s degree in Computer Science, Information Technology, or related field.
- 5+ years of experience with Azure and Databricks architecture, migration, and development.
- Strong experience in building and optimizing data pipelines, architectures, and data sets.
- Proficiency in programming languages such as Python, Scala, or Java.
- Experience with big data tools and technologies (Spark, Hadoop, etc.).
- Solid understanding of SQL and relational databases.
- Experience with cloud infrastructure management and monitoring tools.
- Excellent problem-solving skills and the ability to troubleshoot complex issues.
- Strong communication skills and the ability to work effectively in a remote team.
- Ability to work in the US CST time zone with up to 6 hours of overlap.
Preferred Skills:
- Familiarity with other cloud platforms and tools.
- Certification in Azure and/or Databricks is a plus.
Technical Skills:
- Azure Data Factory (ADF)
- Databricks
- SQL and NoSQL databases
- Data modeling and data warehousing
- ETL processes and tools
- Azure Synapse Analytics
- Python and/or Scala
- Power BI or other data visualization tools