Lead Data Engineer

Lead Data Engineer required by our market leading, award winning, retail sector client based in Avonmouth. Our client have recently migrated across to Microsoft Dynamics Business Central and require a Lead Data Engineer to lead the development, maintenance, support, and integration across an increasing array of data sources into an Azure-hosted SQL data warehouse and the Power BI platform.

This is a hybrid role with 2-3 days a week onsite but with flexibility.

The Lead Data Engineer will oversee the technical aspects of Reporting & BI solutions within the Technology team, providing robust reporting capabilities to support Commercial, Operational, and Finance functions.

Main Responsibilities

• Analyse, optimise, and refactor existing stored procedures, ETL jobs, and data workflows to improve performance and reduce load times on the current data warehouse.
• Clean up and enhance the efficiency of the current Power BI Premium (P1) instance, ensuring optimal resource allocation and utilisation as data volumes and complexity grow.
• Review and maintain the scheduling of data jobs to ensure timeliness and reliability, improving operational processes to meet evolving business needs.
• Implement data quality checks, governance, and performance monitoring for both ETL processes and reporting outputs.
• Lead and actively execute the long-term migration of the on-premises data warehouse to Microsoft Fabric, including hands-on development and implementation of a robust strategy for transitioning datasets, ETL pipelines, and reporting solutions to the cloud.
• Build scalable ETL processes using Azure Data Factory, Azure Synapse, and other Fabric-native tools, ensuring a smooth integration between the on-prem and future cloud environments.
• Design and implement Azure-based data architecture, leveraging tools like Azure Data Lake, Azure Synapse Analytics, and Fabric’s data capabilities to future-proof the platform.
• Utilise Azure tools such as Synapse Analytics, Logic Apps, and Azure Functions to streamline data processing and orchestration across the business.
• Empower stakeholders to build their own reports and dashboards by enhancing data accessibility and developing self-service capabilities.

Required Background

• Extensive experience in maintaining and optimising on-premises SQL Server data warehouses, focusing on stored procedures, indexing, partitioning, and load performance.
• Deep understanding of Azure cloud tools such as Azure Synapse Analytics, Azure Data Lake, Azure Data Factory, and Logic Apps
• Hands-on experience with Microsoft Fabric or similar modern cloud platforms, with a strong focus on data migration strategies and ETL development
• Proficiency in writing advanced SQL queries, stored procedures, and Python scripts for data manipulation, pipeline development, and automation
• Experience with ETL tools (both on-prem like SSIS, and cloud-based like Azure Data Factory) for creating, optimising, and maintaining data pipelines
• Skilled in Power BI Premium, including data modelling, DAX, performance tuning, and advanced reporting features like composite models and incremental refresh
• Strong understanding of data governance, security best practices, and compliance requirements (GDPR, etc.)
• Experience working with Agile methodologies
• Excellent communication skills, capable of collaborating with both technical and non-technical stakeholders to translate business needs into effective data solutions.

The 'Nice to Haves’

• Experience with Microsoft Dynamics 365 Business Central
• Experience of defining and operating within data management best practice frameworks
• Experience working in a Retail environment

This fantastic opportunity comes with a competitive starting salary which is accompanied by the following benefits: 7-10% annual bonus, Health Cash Plan, Life Assurance, 25 days annual leave and a generous staff discount to name but a few.