Job Overview
We are seeking a highly skilled Data Engineer to join our team.
The ideal candidate will have extensive experience working as a Data Engineer, with a strong focus on AWS technologies and serverless architectures.
* Design and implement an AWS Serverless DataLake architecture to efficiently handle large volumes of data and support various data processing workflows.
* Develop data ingestion pipelines and data integration processes, ensuring the smooth and reliable transfer of data from various sources into the DataLake.
* Implement data transformation and data enrichment processes using AWS Lambda, Glue, or similar serverless technologies to ensure data quality and consistency.
Key Responsibilities:
* Collaborate with data scientists and analysts to understand their data requirements and design appropriate data models and schemas in the DataLake.
* Optimize data storage and retrieval mechanisms, leveraging AWS services such as S3, Athena, Redshift, or DynamoDB, to provide high-performance access to the data.
* Monitor and troubleshoot the DataLake infrastructure, identifying and resolving performance bottlenecks, data processing errors, and other issues.