Ampersand Consulting
Senior Data Engineer (Cloud, Azure, SQL, Python, Java)
The UK's leading provider of IT solutions and services is looking to hire an experienced Senior Data Engineer (Cloud, Azure, SQL, Python, Java) to join their rapidly expanding Data Analytics team working closely with the Chief Data Scientist based in Central London.
The Senior Data Engineer (Cloud, Azure, SQL, Python, Java) will implement complex multi/hybrid cloud based big data projects with a focus on collecting, parsing, managing, analyzing and visualizing large sets of data to turn information into insights using multiple technology platforms.
The Senior Data Engineer (Cloud, Azure, SQL, Python, Java) will be a technical contributor with hands-on knowledge of all phases in building large-scale cloud based distributed data processing systems and applications.
Responsibilities for the Senior Data Engineer (Cloud, Azure, SQL, Python, Java)
- Lead the design, implementation, and continuous delivery of pipelines using distributed Azure based big data technologies supporting data processing initiatives across batch and streaming datasets
- Experience in hybrid cloud solutions and experience in GCP resources
- Responsible for development using Scala, Python languages and Big Data Frameworks such as Spark, EMR, Kafka, Storm, Jenkins, Jfrog Artifactory and DataBricks
- Provide administrative support on deployed Azure platform components
- Identify, evaluate and implement cutting edge big data pipelines and frameworks required to provide requested capabilities to integrate external data sources and APIs
- Review, analyse and evaluate market requirements, business requirements and project briefs in order to design the most appropriate end-to-end technology solutions
- Process and manage high volume real time customer interaction streams
- Provide architectural support by building Proof of Concepts & Prototypes
- Self-Starter to deliver data engineering solutions to optimize both the cost and existing solution
Requirements for the Senior Data Engineer (Cloud, Azure, SQL, Python, Java)
- Extensive Software Industry experience
- Development experience with Azure services (such as Data Factories, Data Lake Storage, SQL Elastic Pools, Data Pipeline, DataBricks, Kubernetes Services, Apache Nifi, jFrog etc)
- Development experience with GCP services (Cloud Storage, Cloud Spanner, BigQuery)
- Experience with Apache Spark and NoSQL Implementation
- Extensive working knowledge in different programming or scripting languages like Scala, Java, Linux, Shell, SQL, Python
- Proficiency working with structured, semi-structured and unstructured data sets including social, web logs and real time streaming data feeds
- Able to tune Big Data solutions to improve performance and end-user experience
- Knowledge on Visualization and Data Science Tools
- Expert level usage with Jenkins, GitHub is preferred
If you would like to apply for the Senior Data Engineer (Cloud, Azure, SQL, Python, Java) role then please click the apply button