SonicJobs Logo
Left arrow iconBack to search
  1. Home
  2. London
  3. Database developer
  4. Lead Data Engineer - SC Cleared - Outside IR35 - 6 Months

Lead Data Engineer - SC Cleared - Outside IR35 - 6 Months

SR2
Posted 6 hours ago, valid for 5 days
Location

London, Greater London EC1R 0WX

Salary

£40,000 - £48,000 per annum

info
Contract type

Full Time

In order to submit this application, a Reed account will be created for you. As such, in addition to applying for this job, you will be signed up to all Reed’s services as part of the process. By submitting this application, you agree to Reed’s Terms and Conditions and acknowledge that your personal data will be transferred to Reed and processed by them in accordance with their Privacy Policy.

Sonic Summary

info
  • The position is for a Lead Data Engineer with active SC clearance, requiring 6+ years of experience in data engineering and a strong technical background in Java-based microservices, AWS Glue, PySpark, SQL, and Athena.
  • Responsibilities include designing and optimizing data pipelines, developing microservices, managing AWS services, and leading a team of data engineers.
  • The role involves collaborating with client stakeholders to translate business requirements into technical solutions and ensuring data quality and governance.
  • Candidates should have advanced SQL skills and experience with cloud technologies, particularly AWS services, as well as strong communication and leadership abilities.
  • The position is hybrid, outside IR35, and offers a competitive salary commensurate with experience.

Lead Data Engineer - SC Cleared - 6 Months - Hybrid - Outside IR35

We are seeking someone with a strong technical background in building Java-based microservices, AWS Glue, PySpark, SQL, and Athena to join our team.In this role, you will design, build, and optimise data pipelines and platforms in a cloud environment.

Due to the nature of the role, active SC clearance is required.

Key Responsibilities:

  • Data Pipeline Development: Design, develop, and maintain scalable data pipelines using AWS Glue, PySpark, and Java-based microservices to process large volumes of structured and unstructured data.
  • Microservices Development: Build and optimize Java-based microservices to support data integration, transformation, and orchestration for real-time and batch data processing.
  • AWS Services Management: Leverage AWS services such as S3, Glue, Athena, Lambda, and Step Functions to build and optimize cloud-native data solutions.
  • SQL and Data Queries: Write complex SQL queries to analyse data and support business intelligence tools, ensuring efficient querying and data retrieval using AWS Athena and other querying tools.
  • Team Leadership: Lead and mentor a team of data engineers, providing technical guidance, setting development standards, and ensuring high performance across the team.
  • Client Stakeholder Engagement: Collaborate with key client stakeholders to understand business requirements, translate them into technical solutions, and ensure the successful delivery of data projects.
  • Data Architecture: Work with data architects to define data models, design data storage solutions, and optimize data integration processes to ensure data quality, performance, and scalability.
  • Data Quality and Governance: Implement data quality controls, monitoring, and governance practices to ensure the integrity and security of data across the platform.
  • Performance Tuning: Identify bottlenecks in data processes and optimize performance by fine-tuning Glue jobs, PySpark code, SQL queries, and microservices.

Required Skills and Experience:

  • Experience: 6+ years of hands-on experience in data engineering, with a focus on building cloud-based data platforms.
  • Java Microservices: Strong expertise in developing and deploying Java-based microservices in a cloud environment.
  • AWS Glue & PySpark: Proven experience with AWS Glue and PySpark to build, schedule, and run ETL/ELT processes at scale.
  • SQL Expertise: Advanced proficiency in SQL for querying and optimizing data in cloud-based environments like AWS Athena or Redshift.
  • Cloud Technologies: Deep understanding of AWS services (S3, Lambda, Glue, Athena, Step Functions) and experience with cloud data infrastructure.
  • Team Leadership: Experience leading a team of data engineers, providing mentorship, technical oversight, and fostering collaboration.
  • Client Interaction: Strong communication skills with experience working directly with client stakeholders to gather requirements and deliver solutions that align with business objectives.

Preferred Skills:

  • Big Data Technologies: Familiarity with big data ecosystems and tools such as Hadoop, Spark, Kafka, or similar.
  • Automation & CI/CD: Experience with CI/CD pipelines, containerization (Docker), and orchestration tools (Kubernetes).
  • Data Governance: Knowledge of data governance frameworks, data security, and compliance practices in cloud environments.
  • Certifications: AWS certifications such as AWS Certified Data Analytics or AWS Certified Solutions Architect are a plus.

Lead Data Engineer - SC Cleared - 6 Months - Hybrid - Outside IR35

Apply now in a few quick clicks

In order to submit this application, a Reed account will be created for you. As such, in addition to applying for this job, you will be signed up to all Reed’s services as part of the process. By submitting this application, you agree to Reed’s Terms and Conditions and acknowledge that your personal data will be transferred to Reed and processed by them in accordance with their Privacy Policy.

Logo
SonicJobs
Your next career move. Sorted.
Search jobsbackground image

Contact us

© 2017 - 2024 SonicJobs App Ltd. All rights reserved

Terms & Conditions | Privacy Policy | AI Agent | Browse Jobs