Job Overview
We're seeking a talented Data Engineer to join our team. The ideal candidate will have expertise in designing and implementing scalable data architectures, as well as developing efficient data ingestion pipelines.
Key Responsibilities:
* Design and Implement Scalable Data Architectures: Develop an AWS Serverless DataLake architecture that can efficiently handle large volumes of data and support various data processing workflows;
* Develop Efficient Data Ingestion Pipelines: Design and implement data ingestion pipelines and data integration processes to ensure the smooth and reliable transfer of data from various sources into the DataLake;
* Implement Data Transformation and Enrichment Processes: Utilize AWS Lambda, Glue, or similar serverless technologies to design and implement data transformation and enrichment processes that ensure data quality and consistency;
* Collaborate with Cross-Functional Teams: Collaborate with data scientists and analysts to understand their data requirements and design appropriate data models and schemas in the DataLake;
* Optimize Data Storage and Retrieval Mechanisms: Leverage AWS services such as S3, Athena, Redshift, or DynamoDB to optimize data storage and retrieval mechanisms, ensuring high-performance access to the data;
* Monitor and Troubleshoot Data Infrastructure: Continuously monitor and troubleshoot the DataLake infrastructure, identifying and resolving performance bottlenecks, data processing errors, and other issues;
* Drive Innovation and Knowledge Sharing: Continuously evaluate new AWS services and technologies to enhance the DataLake architecture, improve data processing efficiency, and drive innovation within the organization;
* Mentor and Train Junior Engineers: Mentor and provide technical guidance to junior data engineers, fostering their growth and ensuring adherence to best practices.