We are seeking a skilled Data Engineer to join our organization. As a Data Engineer, you will be responsible for designing and implementing AWS serverless architectures, working with large-scale data processing systems, and collaborating with cross-functional teams.
Key Responsibilities:
* Data Engineering: Design and implement an AWS Serverless DataLake architecture to efficiently handle large volumes of data and support various data processing workflows;
* Data Ingestion: Develop data ingestion pipelines and data integration processes, ensuring the smooth and reliable transfer of data from various sources into the DataLake;
* Data Transformation: Implement data transformation and data enrichment processes using AWS Lambda, Glue, or similar serverless technologies to ensure data quality and consistency;
* Collaboration: Collaborate with data scientists and analysts to understand their data requirements and design appropriate data models and schemas in the DataLake;
* Data Storage: Optimize data storage and retrieval mechanisms, leveraging AWS services such as S3, Athena, Redshift, or DynamoDB, to provide high-performance access to the data;
* Troubleshooting: Monitor and troubleshoot the DataLake infrastructure, identifying and resolving performance bottlenecks, data processing errors, and other issues;
* Professional Development: Continuously evaluate new AWS services and technologies to enhance the DataLake architecture, improve data processing efficiency, and drive innovation;
* Mentorship: Mentor and provide technical guidance to junior data engineers, fostering their growth and ensuring adherence to best practices;
* Cross-Functional Collaboration: Collaborate with cross-functional teams to understand business requirements, prioritize tasks, and deliver high-quality solutions within defined timelines.