Lead Data Engineer

Cynosure Corporate Solutions

8 - 12 Years
3 Openings
Not Disclosed
Chennai
Posted 22 days ago
Telephonic interview

Key Skills

Spark Hadoop Eco Systems Data Streaming Pipelines Python Scala Java Sqoop

Apply

Job Alert Similar Jobs Share

Job Description

Key responsibilities:

Working with clients to understand their data.

Basedontheunderstanding you will be building the data structures and pipelines.

You will be working on the application from end to end collaborating with UI and other development teams.

You will be responsible for building the data pipelines to migrate and load the data into the HDFSeither on-prem or in the cloud.

Developing Data ingestion/process/integration pipelines effectively.

Creating Hive data structures,metadata and loading the data into data lakes / Big Data warehouse environments.

Optimized(Performance tuning) many data pipelines effectively to minimize cost.

Codeversioning control and git repository is up to date.

You will be responsible for building and maintaining CI/CD of the data pipelines.

You will be managing the unit testing of all data pipelines

Skills & Experience:

Bachelor’s degree in computer science or related field.

Minimum of 5+years working experience with Spark, Hadoop eco systems.

Minimum of 4+years working experience on designing data streaming pipelines.

Should be an expert in either Python/Scala/Java.

Should have experience in Data Ingestion and Integration into data lake using hadoop ecosystem tools such as Sqoop, Spark, SQL, Hive, Airflow, etc..

Should have experience optimizing (Performance tuning) data pipelines.

Minimum experience of 3+ years on NoSQL and Spark Streaming.

Knowledge of Kubernetes and Docker is a plus.

Should have experience with Cloud services either Azure/AWS.

Should have experience with on-prem distribution such as Cloudera/HortonWorks/MapR.

Basic understanding of CI/CD pipelines.

Basic knowledge of Linux environment and commands.

Experience
8 - 12 Years
No. of Openings
3
Education
Professional Degree
Role
Lead Data Engineer
Industry Type
IT-Hardware & Networking / IT-Software / Software Services
Gender
[ Male / Female ]
Job Country
India
Type of Job
Full Time
Work Location Type
Work from Office

About Cynosure Corporate Solutions

Cynosure Corporate Solutions is a name that has been winning the trust of clients for years through its organized operations. We are providing recruitment and consulting services under one roof. Ours is a Chennai (Tamil Nadu, India) based human resource management company, specialized in providing services of a placement agency including HR consultant, training services, domestic placement and recruitment consultant. In addition to this, we provide career consultancy to candidates and services for work at home. Also, we render solutions for human capital management through a scientific approach. Owing to our well-organized database of the technical & non-technical job-openings and job-seekers, we help organizations in finding suitable talent.