The opportunity
As a Data Engineer, You will be responsible for the creation, schedule and optimization of the data processing pipelines using Apache Spark and Apache Airflow deployed on our data analytics platform.
The Data Engineer will support our software developers, architects, and DevOps team on data initiatives. Ability to work in an Agile / Scrum model.
Your key responsibilities
- Create, build and deploy data pipelines that clean and process data for analysis
- Build reusable component pipelines that can be assembled for more complex pipelines
- Identify, design, and implement internal process improvements : automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability
- Work with structured and unstructured data sets
- Work with QA to develop an automation test strategy for data pipelines that aligns with the platform’s architecture
Skills and attributes for success
Minimum 7 years hands-on experience building complex data pipelinesStrong experience in data processing and optimization using Apache SparkGood knowledge of data manipulation and transformationExperience with Apache Airflow, Python, SQLDeployment and provisioning automation tools (, Docker, Kubernetes, CI / CD)Hands-on experience managing and further developing distributed systems and clusters for both batch as well as streaming data (S3 / Spark and / or Kafka / Flink)Hands-on experience in training and evaluating ML modelsExperience with GolangExperience with monitoring and observability (ELK stack)Bash scripting and Linux systemsExperience with buildpacksSecurity, authentication, and authorisation (LDAP / Kerberos / PAM)Experience working in cloud environment (, GCP)Ideally, you’ll also have
Experience with data warehousing solutionsKnowledge of big data tools and technologiesStrong problem-solving skills and attention to detailExcellent communication and collaboration skillsAbility to work independently and as part of a teamWhat we offer
EY Global Delivery Services (GDS) is a dynamic and truly global delivery network. We work across ten locations – Argentina, China, Hungary, India, the Philippines, Poland, Sri Lanka, Mexico, Spain and the United Kingdom – and with teams from all EY service lines, geographies and sectors, playing a vital role in the delivery of the EY growth strategy. From accountants to coders to advisory consultants, we offer a wide variety of fulfilling career opportunities that span all business disciplines. In GDS, you will collaborate with EY teams on exciting projects and work with well-known brands from across the globe. We’ll introduce you to an ever-expanding ecosystem of people, learning, skills and insights that will stay with you throughout your career.
Continuous learning : You’ll develop the mindset and skills to navigate whatever comes next.Success as defined by you : We’ll provide the tools and flexibility, so you can make a meaningful impact, your way.Transformative leadership : We’ll give you the insights, coaching and confidence to be the leader the world needs.Diverse and inclusive culture : You’ll be embraced for who you are and empowered to use your voice to help others find theirs.About EY
EY | Building a better working world