Data Engineer – London, South East England
The Data Engineer will provide technical expertise in the analysis, design, development, rollout and maintenance of data integration initiatives. This role will contribute to implementation methodologies and best practices, as well as work on project teams to analyse, design, develop and deploy business intelligence/data integration solutions to support a variety of customer needs. Provides technical expertise in needs identification, data modelling, data movement and transformation mapping (source to target), automation and testing strategies, translating business needs into technical solutions with adherence to established data guidelines and approaches from a business unit or project perspective whilst leveraging best fit technologies (e.g., cloud, Hadoop, NoSQL, etc.) and approaches to address business and environmental challenges. Effectively partners with the client team.
Leadership not only in the conventional sense but also within a team we expect people to be leaders. The candidate should elicit leadership qualities such as Innovation, Critical thinking, optimism/positivity, Communication, Time Management, Collaboration, Problem-solving, Acting Independently, Knowledge sharing and Being Approachable.
Essential Duties:
- Design, develop, test, and deploy data integration processes (batch or real-time) using tools such as Microsoft SSIS, Azure Data Factory, Databricks, Matilion, Airflow, Sqoop, etc.
- Create functional & technical documentation – e.g. ETL architecture documentation, unit testing plans and results, data integration specifications, data testing plans, etc.
- Provide a consultative approach with business users, asking questions to understand the business need and deriving the data flow, conceptual, logical, and physical data models based on those needs. Perform data analysis to validate data models and to confirm the ability to meet business needs.
- Stays current with emerging and changing technologies to best recommend and implement beneficial technologies and approaches for Data Integration
- Design and develop enterprise self-service analytic solutions, business intelligence reports, visualisations and best practice standards. Toolsets include but are not limited to SQL Server Analysis and Reporting Services, Microsoft Power BI, Tableau and Qlik.
- Work with report team to identify, design and implement a reporting user experience that is consistent and intuitive across environments, across report methods, defines security and meets usability and scalability best practices.
Education & Experience:
- 5 Years of industry implementation experience with data integration tools such as Microsoft SSIS, Azure Data Factory, Databricks, Glue, Step Functions, Airflow, Apache Flume/Sqoop/Pig, etc.
- 1-3 years of consulting experience preferred
- Bachelor’s degree or equivalent experience, Master’s Degree Preferred
- Strong data warehousing, OLTP systems, data integration and SDLC
- Experience in big data frameworks & working experience in Spark or Hadoop or Hive (incl. derivatives like pySpark (prefered), SparkScala or SparkSQL) or Similar, along with experience in libraries/frameworks to accelerate code development
- Experience using major data modelling tools (examples: ERwin, ER/Studio, PowerDesigner, etc.)
- Experience with major database platforms (e.g. SQL Server, Oracle, Azure Data Lake, Hadoop, Azure Synapse/SQL Data Warehouse, Snowflake, Redshift etc.)
- Strong experience in orchestration & working experience in either Data Factory or HDInsight or Data Pipeline or Cloud composer or Similar
- Understanding and experience with major Data Architecture philosophies (Dimensional, ODS, Data Vault, etc.)
- Understanding of modern data warehouse capabilities and technologies such as real-time, cloud, and Big Data.
- Understanding of on-premises and cloud infrastructure architectures (e.g. Azure, AWS, GCP)
- Experience in Agile Process (Scrum cadences, Roles, deliverables) & working experience in either Azure DevOps, JIRA or Similar with Experience in CI/CD using one or more code management platforms
- 1-3 years of development experience in decision support/business intelligence environments utilizing tools such as SQL Server Analysis and Reporting Services, Microsoft’s Power BI, Tableau, looker etc.
