- We are seeking a Senior Data Engineer to join our Data & Analytics team. The ideal candidate will design, build, and maintain robust on-premise data pipelines, architect flexible lakehouse solutions, and lead data infrastructure initiatives to enable data-driven decision-making across the organization.
Mandatory Skill(s)
- Bachelor's degree in Computer Science, IT, Engineering, or related field.
- Must have 5 years of hands-on experience in data engineering and data pipeline development;
- Hands on experience in SQL, with in-depth expertise in Microsoft SQL Server, including query optimization, indexing strategies, performance tuning, and troubleshooting;
- Hands-on experience with at least one NoSQL database, such as MongoDB or Apache Cassandra;
- Practical experience in distributed data processing using Apache Spark and PySpark, including performance optimization and large-scale data processing;
- Strong Python programming skills for data processing, automation, and developing scalable, reliable, production-grade data pipelines;
- Expertise in designing, deploying, and managing containerized data pipelines using Kubernetes in on-premises environments;
- Strong understanding of relational and non-relational data modeling concepts, including schema design and optimization;
- Hands-on experience with Apache Airflow or similar orchestration tools for scheduling, monitoring, and managing batch and real-time data pipelines;
- Knowledge of data security and governance practices, including role-based access control (RBAC), data masking, and compliance with relevant regulatory and organizational standards;
Desirable Skill(s)
Good to have experience with on-premise data virtualization or logical data warehouse concepts.
Good to have real-time streaming technologies (Kafka, Apache Flink);
Responsibilities
- Design and develop autonomous, production-grade ETL/ELT data pipelines using Python and PySpark that ingest, transform, and deliver high-quality data while maintaining integrity and performance standards.
- Implement and manage flexible lakehouse architecture across raw, curated, and consumption layers, including data partitioning, cataloging, and metadata management;
- Deploy and manage data pipelines using Kubernetes and Docker to ensure scalability, reliability, and efficient resource utilization in on-premise environments;
- Leverage strong SQL Server expertise to design optimal data models, write complex queries, and perform query optimization across the data platform;
- Establish and maintain robust CI/CD practices for data pipeline deployment, including automated testing, version control, and continuous monitoring;
- Enforce security, governance, and role-based access controls across all data layers while ensuring compliance and auditability;
- Mentor junior engineers, conduct code reviews, and establish best practices across the team;
- Collaborate with Data Scientists, Business Analysts, and stakeholders to deliver datasets aligned with operational and analytical needs;
- Provide L3 support and expert consultation for complex data challenges; evaluate and recommend new tools and practices to improve agility and performance.
If you are interested in this role, click on the “Apply to this job” button below or you could also write in with your CV to Binodini Panda at binodini.p@sciente.com quoting the job title.
