We are looking for a Data Engineer on a 12 month Fixed term,  to join our growing data team at Irwin Mitchell. This role is key to accelerating data value creation and enabling improved data-driven decisions and advanced analytics across the firm.

You will design, develop, and maintain data engineering solutions that support our transition to a modern cloud data platform. Working with technologies such as Azure Data Factory, Azure Databricks, and PySpark, you will help build and optimize data pipelines and lakehouse solutions that ensure data is accessible, reliable, and secure.

This is an exciting opportunity to contribute to creating an aspirational data culture, leveraging industry best practices and modern technologies.

Day to day 

• Design, develop, and maintain end-to-end data pipelines and lakehouse solutions using Python/PySpark, Azure Data Factory, and Azure Databricks.
• Collaborate with cross-functional teams to understand data requirements and translate them into efficient data models and ETL processes.
• Implement data integration and transformation workflows to extract, clean, and load data from various sources.
• Optimize data pipelines for performance, scalability, and reliability using distributed computing techniques.
• Monitor and troubleshoot data pipelines, ensuring data integrity, availability, and security.
• Stay up-to-date with advancements in data engineering and Azure technologies, recommending new tools and frameworks to enhance capabilities.

Essential 

• Proven experience as a Data Engineer with strong focus on Python/PySparkand Azure technologies.
• Hands-on experience with Azure Data Factory, Azure Databricks, Azure SQL Database, Azure Data Lake Storage, and related services.
• Understanding of data modelling, ETL development, and data integration techniques.
• Knowledge of cloud-based data architectures, distributed computing, and big data technologies.
• Experience with Terraform/Bicepfor infrastructure-as-code and deployments via Azure DevOps.
• Strong communication and Stakeholder managment experience