PySpark Developer

PySpark Developer

Description

We are looking for a skilled Data Engineer with expertise in Python, PySpark, and Cloudera to join our team. The ideal candidate will be responsible for developing and optimizing big data pipelines while ensuring efficiency and scalability. Experience with Databricks is a plus. Additionally, familiarity with Git, GitHub, Jira, and Confluence is highly valued for effective collaboration and version control.

Key Responsibilities

• Design, develop, and maintain ETL pipelines using Python and PySpark.
• Work with Cloudera Hadoop ecosystem to manage and process large-scale datasets.
• Ensure data integrity, performance, and reliability across distributed systems.
• Collaborate with data scientists, analysts, and business stakeholders to deliver data-driven solutions.
• Implement best practices for data governance, security, and performance tuning.
• Use Git and GitHub for version control and efficient code collaboration.
• Track and manage tasks using Jira, and document processes in Confluence.
• Optional) Work with Databricks for cloud-based big data processing.

Required Skills & Experience

Preferred Qualifications