Lead Data Engineer - MHCLG - G7

Location

Darlington, London, Manchester, Wolverhampton

About the job

Job summary

Here at the Ministry of Housing, Communities & Local Government (MHCLG), we work on things that make a real difference to people’s lives.  

Whether it's through the homes we live in, the work of our local councils, or the communities we’re all part of, our work is at the top of the political agenda. We have ambitious and far-reaching outcomes to achieve this year and, if you’re thinking of joining us, there’s never been a more exciting time.  

We have 3,500+ staff who are based in 20 offices across the UK.  

AI technology is rapidly evolving and there is immense opportunity to harness this capability to transform delivery at MHCLG. We are looking for a passionate and enthusiastic lead data engineer to join our team and be at the leading edge of AI delivery in a central government department, to deliver high-impact projects at pace to improve outcomes and productivity.  

The new AI directorate at MHCLG will drive forward AI development and adoption across the department and support partners in local government. As a lead data engineer, you will unlock new approaches through the adoption of AI tools and techniques to help enable faster and more informed decision making. You will be motivated, flexible, enthusiastic and passionate about utilising data and AI responsibly for the public good. 

We particularly welcome candidates from an ethnic minority background and other underrepresented groups to apply, as we work to continually improve our ability to represent the places and communities we support through our work. 

You will join a multi-disciplinary team of technical experts (including data engineers, data scientists, developers, delivery managers, service owners) with an ambitious and exciting remit to harness the potential of AI, working on high impact projects that deliver operational efficiencies and support policy development through the rapid prototyping and scaling of products and services. You will work collaboratively across the organisation, interacting with a range of stakeholders to explore, build and maintain foundational AI capability and champion the role and impact of data engineering in the process.

Find out more about what it's like to work in a digital, data and technology role at MHCLG  including our culture, ways of working, career progression and staff benefits. 

Job description

As the Lead Data Engineer, you’ll:  

work closely with data scientists, developers, and the wider team of DDaT expertise to develop robust, repeatable processes for managing and automating routine and repetitive tasks for data flows between teams 

lead workstreams to identify and deliver data pipelines for AI prototyping and scaling, applying knowledge of systems integration 

identify areas of innovation in data tools and techniques and recognise appropriate timing for adoption 

support and enable effective data management, ensuring documentation, data dictionaries, and metadata are readily available across a range of stakeholders 

confidently define and build robust data models and ETL processes to support AI delivery in line with best practice and standards 

establish enterprise-scale data integration procedures across the data development life cycle to ensure decisions and actions are based upon reliable and accurate data 

continuously improve delivery by identifying potential efficiencies and savings in existing processes, ensuring resources are utilised appropriately. Lead on collaboration and building capability while supporting the continuous development and upskilling of the wider technical team 

communicate complex technical issues clearly and concisely to the team and to senior leaders to aide decision-making. Champion data engineering across the organisation and more widely across government 

Person specification

As the Lead Data Engineer, you’ll: 

understand the concepts and principles of data modelling and lead on the delivery of relevant data models 

have extensive working knowledge and experience with ETL processes and data integration, including the use of APIs, data exchange with numerous systems, SQL experience and technical database expertise 

have the ability to build data pipelines using cloud technologies such as Azure Data Factory/Databricks and be experienced writing parameterised code such as SQL/PL-SQL and/or Python/Spark 

be able to develop integration procedures across all stages of the data development life cycle while continually improving processes and efficiency and be able to identify issues in data pipelines and manage their resolution 

utilise key stakeholder management skills to effectively manage expectations, actively listen and interpret business needs, communicate progress, and navigate challenging conversations, all while fostering a service orientated culture around data 

have strong experience working with cloud data storage (e.g. Azure blob storage / data lake), ideally involving the development and update of new and existing data sources across multiple environments (sandbox / test / production) 

apply knowledge of standard development practices including code versioning and reviews, and ensuring coding standards are applied and familiarity using CI/CD / DevOps/ Github repository for versioning, code promotion and change management 

have experience of defining and documenting data workflows across systems and operations, including capturing and documenting complex data definitions and business rules, applying data governance and metadata principles where appropriate 

Behaviours

We'll assess you against these behaviours during the selection process:

Seeing the Big Picture
Making Effective Decisions
Communicating and Influencing
Working Together
Managing a Quality Service