Data Engineer - ONS - SEO

The Office for National Statistics (ONS) is the UK’s largest producer of official statistics, covering a range of key economic, social and demographic topics. These include measuring changes in the value of the UK economy, estimating the size, geographic distribution, and characteristics of the population, and providing indicators of price inflation, employment, earnings, crime, and migration. 

You'll be responsible for managing our Python codebase, creating intuitive user interfaces, and connecting to cloud databases, helping to ensure our systems remain efficient and safe.

Job description

You'll have a passion for combining data engineering principles with software development to create data products that enhance our ability to process, store, and retrieve information across various platforms. You will work closely with a multidisciplinary team of developers, engineers, and production staff, ensuring the smooth and efficient development and deployment of solutions.

Responsibilities 

Design, develop, and maintain ETL pipelines and data processing workflows to manage and manipulate datasets, ensuring data quality, consistency, and accuracy.
Create intuitive and user-friendly interfaces for internal production teams, making sure that all software products are highly usable and meet the users’ requirements.
Manage and enhance the Python codebase used for both data processing and user interface functionality, ensuring clean, maintainable, and well-documented code.
Build and maintain scalable, secure, and efficient cloud-based solutions, including designing database schemas and managing data flows.
Implement software development best practices, including version control (Git), continuous integration, testing, and deployment (CI/CD).
Troubleshoot and optimise existing data pipelines and software, ensuring performance improvements and resilience in production environments.
Collaborate with data scientists, analysts, and other teams to ensure efficient data integration, storage, and processing, and the transfer of appropriate data engineering knowledge.
Provide line management and support to junior team members, guiding their growth and development.

Person specification

Essential Skills Criteria:

Ability to determine customer requirements and translate these into delivery design for data processing.
Ability to design, write, and operate ETL pipelines, in the context of distributed processing, applying coding, data, and documentation standards, in the language required by the business area.
Experience of developing in one of the data platforms ONS use; Cloudera or GCP. With experience in Python and knowledge of SQL and/or R would be an advantage.
Experience of writing clean, efficient and well documented code for data processing tasks to a specification.
Experience of using and applying code version control across all work using Git with GitHub or GitLab.
Ability to lead the development of ETL pipelines, including effective delegation of work.
Good understanding of the principles of data processing in a distributed system, and ability to use this understanding to ensure and design robust coding in a distributed environment, including the creation of flow charts to illustrate coding flows and unit tests and integration testing.

Behaviours

We'll assess you against these behaviours during the selection process:

Leadership
Changing and Improving
Managing a Quality Service
Working Together

Technical skills

We'll assess you against these technical skills during the selection process:

Programming and Build
Data Integration Design