Data Engineer

Data Engineer

Location: Primarily remote, with 1- 2 days per week in-office in LondonDuration: 12-month contract (with potential for extension)

ONLY CANDIDATES WITH PREVIOUS FINANCIAL SERVICES EXPERIENCE WILL BE CONSIDERED

Overview:Our client, a respected Financial Services organisation, is embarking on a large-scale data transformation initiative and is seeking an experienced Data Engineer to join their team. This role presents an opportunity to work on a cutting-edge data platform within a financial services environment.

The successful applicant will play a key role in designing, developing, and optimizing data pipelines and architectures using the full Azure stack. They will contribute to the build-out of Data Lakes and data platforms, ensuring high performance, scalability, and robust data governance. 

• Design, develop, and optimize scalable data pipelines using Azure Databricks and Spark (Scala, Python, SQL).

• Implement and maintain Delta Lake architectures for structured and unstructured data processing.

• Develop ETL/ELT frameworks for efficient data ingestion into Azure Data Lake.

• Implement robust Data Governance frameworks, focusing on Metadata, Data Quality, Lineage, and Data Access Models.

• Utilize Unity Catalog (or similar Data Governance tools) to manage and control data assets effectively.

• Leverage MS Purview or other metadata/data quality tools such as Collibra, Informatica (MDM, Axon, Data Quality), etc..

• Define and implement Data Modelling best practices to support Data Products and Data Domains.

• Work closely with business stakeholders and data analysts to understand data needs and develop tailored solutions.

• Ensure compliance with regulatory and security standards for handling financial data within a corporate banking environment.

• Support and contribute to the broader data architecture strategy within the organization.

• Expertise in Azure Data Engineering, including:

  • Databricks (Workflows, Jobs, Notebooks, and Pipelines)

  • Azure Data Lake Storage (ADLS)

  • Azure Synapse Analytics (nice to have)

  • Azure Functions & Logic Apps for automation

  • Azure DevOps (CI/CD for data pipelines)

• Strong Spark programming skills using Scala, Python, and SQL.

• Experience with Delta Lake for ACID-compliant data lake architectures.

• Proficiency in data governance tools, including:

  • Unity Catalog (or similar Data Governance/Metadata Management tools)

  • MS Purview (or alternatives like Collibra, Informatica, etc.)

• Understanding of financial services data landscapes, particularly within corporate banking.

• Experience designing scalable and efficient ETL/ELT frameworks.

• Solid understanding of Data Modelling principles and how they relate to Data Products and Data Domains.

• Experience in Data Architecture is a strong advantage.

• Extensive experience in Data Engineering within large-scale transformation projects.

• Experience creating Data Governance frameworks / structures alongside a Data Transformation.

• Previous experience working within Financial Services is essential.

• Strong analytical and problem-solving skills with a practical approach to data challenges.

• Excellent communication and stakeholder management skills.

• Self-starter mindset, capable of working independently while collaborating effectively with teams.

• Ability to navigate complex data ecosystems and drive efficiency and innovation.

This is a fantastic opportunity for a Data Engineer with strong Azure expertise and a background in financial services to contribute to a strategic transformation program.