Data Engineer Position
We are seeking an experienced data engineer to join our team. As a data engineer, you will be responsible for designing and implementing scalable data processing systems using AWS technologies.
Key Responsibilities:
* Design and implement an AWS Serverless DataLake architecture to efficiently handle large volumes of data and support various data processing workflows.
* Develop data ingestion pipelines and data integration processes, ensuring the smooth and reliable transfer of data from various sources into the DataLake.
* Implement data transformation and data enrichment processes using AWS Lambda, Glue, or similar serverless technologies to ensure data quality and consistency.
* Collaborate with data scientists and analysts to understand their data requirements and design appropriate data models and schemas in the DataLake.
* Optimize data storage and retrieval mechanisms, leveraging AWS services such as S3, Athena, Redshift, or DynamoDB, to provide high-performance access to the data.
* Monitor and troubleshoot the DataLake infrastructure, identifying and resolving performance bottlenecks, data processing errors, and other issues.
* Continuously evaluate new AWS services and technologies to enhance the DataLake architecture, improve data processing efficiency, and drive innovation.
* Mentor and provide technical guidance to junior data engineers, fostering their growth and ensuring adherence to best practices.
* Collaborate with cross-functional teams to understand business requirements, prioritize tasks, and deliver high-quality solutions within defined timelines.
Requirements:
* 5+ years of experience working as a Data Engineer, with a strong focus on AWS technologies and serverless architectures.
* Experience working as a Data Engineer, with focus on Azure is valuable.
* In-depth knowledge of AWS services such as S3, Lambda, Glue, Athena, Redshift, and DynamoDB, and their capabilities for building scalable data processing systems.
* Proven expertise in designing and implementing AWS serverless architectures for large-scale data processing and storage.
* Strong programming skills in languages like Python, Java, or Scala, along with experience using SQL for data manipulation and querying.