Sr. Data Engineer

TekIT Software Solutions Pvt. Ltd. (India & USA) · Bengaluru, India

Data Engineer – Data Platform

Location: Bangalore

Experience: 5+ Years

Role Overview

We are looking for a hands-on Data Engineer to design, develop, and optimize scalable data platforms and data pipelines . The ideal candidate will have strong expertise in Python, SQL, PySpark/Spark, Databricks , and modern data engineering practices.

Candidates should have hands-on experience with at least one cloud platform – AWS, Azure, or GCP . Experience with BigQuery, Lakehouse architecture, Delta Lake, Airflow, Kafka, and CDC will be highly valued.

Key Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines using Python, SQL, PySpark, and Apache Spark .
• Build and optimize data processing solutions using Databricks and cloud-native data services.
• Develop batch and real-time data ingestion pipelines using technologies such as Airflow, Kafka, CDC , and other streaming/integration frameworks.
• Implement modern Lakehouse and Medallion Architecture using Delta Lake or similar technologies.
• Develop reliable data pipelines for structured and semi-structured data from multiple sources.
• Implement data quality, validation, reconciliation, monitoring, logging, and error-handling mechanisms.
• Optimize Spark jobs and data pipelines for performance, scalability, reliability, and cloud cost efficiency .
• Work with cloud storage, compute, databases, and data services across AWS, Azure, or GCP .
• Implement engineering best practices including Git, CI/CD, unit testing, integration testing, and code reviews .
• Troubleshoot data pipeline failures and resolve performance and data-quality issues.
• Collaborate with Data Architects, Solution Architects, Analysts, Developers, and Business Stakeholders to understand requirements and deliver scalable data solutions.
• Contribute to technical design, documentation, coding standards, and continuous improvement of the data platform.

Must-Have Skills

• 5+ years of hands-on experience in Data Engineering .
• Strong programming skills in Python .
• Strong expertise in SQL and data manipulation.
• Hands-on experience with PySpark / Apache Spark .
• Strong hands-on experience with Databricks .
• Good understanding of ETL/ELT concepts, data pipelines, data modeling, and distributed data processing .
• Experience with Lakehouse Architecture, Medallion Architecture, and Delta Lake .
• Hands-on experience with at least one cloud platform: AWS / Azure / GCP .
• Experience with one or more cloud data/storage services such as:
• AWS: S3, Glue, EMR, Redshift
• Azure: ADLS, Data Factory, Synapse
• GCP: GCS, BigQuery, Dataflow
• Exposure to Apache Airflow or other workflow orchestration tools.
• Experience with Kafka, CDC, or real-time/streaming data pipelines .
• Good understanding of data quality, monitoring, testing, and pipeline optimization .
• Experience with Git and CI/CD pipelines .
• Strong analytical, troubleshooting, and problem-solving skills.
• Good communication and collaboration skills.

Good-to-Have Skills

• Apache Iceberg / BigLake
• dbt
• Databricks Unity Catalog
• Kafka / Structured Streaming
• Change Data Capture (CDC) tools
• Infrastructure as Code (IaC) – Terraform or similar
• Cloud security, IAM, networking, and access management
• Cloud cost optimization
• Data governance and metadata management
• Experience working with data warehouses and dimensional data modeling

Preferred Technical Stack