We are in pursuit of an Astute Data Engineer to develop and streamline Data & AI products. The successful candidate will architect and build ingestion pipelines to consolidate data from different systems.
Mandatory Skill(s)
- Bachelors Degree in Computer Science, Engineering or any relevant field;
- At least 5 years of experience in data engineering and modelling;
- Proficiency in Python, JavaScript, or Scala (at least one required), with a strong aptitude for learning;
- Familiarity with ETL systems – monitoring, and performance tuning; diagnose issues and deploy measures to prevent recurrence;
- Demonstrated ability to build software for developing products like DataHive, FDP (Fraud detection platform);
- Experience with Opensource or Cloud platforms to maintain data catalogue to document data assets & metadata;
- Good understanding in using cloud technologies such as AWS, Azure, and Google Cloud;
- Experience with distributed data technologies such as Spark, Hadoop;
- Experience with orchestration frameworks such as Azure Data Factory, Airflow;
- Proficiency in programming languages such as Python, R, Java, or Scala;
- Ability to solve abstract problems through coding;
- Effective code communication and reasoning with engineers;
- Demonstrated ability in writing efficient code;
- Proactive initiative and a strong passion for software engineering.
Desirable Skill(s)
- Exposure to cloud or on-prem production environments and deployment;
- Familiarity with building and using CI/CD pipelines;
- Familiarity with DevOps tools such as Git, Terraform, Git;
- Experience with Databricks or Snowflake.
Responsibilities
- Establish and Architect efficient data ingestion pipelines, ensuring data is collected, cleaned, integrated, and harmonized from diverse sources;
- Conduct daily monitoring and maintenance of databases and ETL systems, including capacity planning, performance tuning, and proactive issue diagnosis, implementing preventative measures to minimize downtime and ensure optimal database availability;
- Develop and test reusable data models to meet end-user analytical and reporting needs, translating business requirements into effective data structures and ensuring data accuracy;
- Develop and implement secure mechanisms for end-users and systems to access data within the data warehouse;
- Research, evaluate, and propose innovative technologies and processes to enhance the data infrastructure, staying abreast of industry trends;
- Collaborate closely with data stewards to establish and enforce comprehensive data governance policies, best practices, and procedures, defining data quality standards;
- Manage a detailed data catalogue, documenting data assets, metadata, and lineage to facilitate data discovery and utilization;
- Implement and monitor rigorous data quality checks and validation procedures, developing quality rules and executing data cleansing and transformation processes;
- Enforce and implement robust data security measures, including access control, encryption, and data masking, to protect sensitive data.
If you are interested in this role, click on the “Apply to this job” button below or you could also write in with your CV to Saraja Dornala at saraja.d@sciente.com quoting the job title.
