Key Responsibilities
The ideal candidate will be responsible for designing and implementing a scalable data architecture on AWS to efficiently handle large volumes of data and support various data processing workflows.
* Data Ingestion: Developing data ingestion pipelines and integration processes to ensure the smooth and reliable transfer of data from various sources into the data lake;
* Data Transformation: Implementing data transformation and enrichment processes using serverless technologies like AWS Lambda and Glue to ensure data quality and consistency;
* Data Modeling: Collaborating with data scientists and analysts to understand their data requirements and design appropriate data models and schemas in the data lake;
* Data Storage and Retrieval: Optimizing data storage and retrieval mechanisms, leveraging AWS services like S3, Athena, and Redshift, to provide high-performance access to the data;
* Infrastructure Monitoring: Monitoring and troubleshooting the data lake infrastructure, identifying and resolving performance bottlenecks, data processing errors, and other issues;
* Technology Evaluation: Continuously evaluating new AWS services and technologies to enhance the data lake architecture, improve data processing efficiency, and drive innovation;
* Mentorship: Mentor junior data engineers, providing technical guidance and ensuring adherence to best practices;
* Cross-Functional Collaboration: Collaborate with cross-functional teams to understand business requirements, prioritize tasks, and deliver high-quality solutions within defined timelines.
About This Role
This role requires a strong understanding of data engineering principles, experience with AWS services, and excellent problem-solving skills. The ideal candidate will have a passion for innovation, a commitment to delivering high-quality solutions, and a desire to work collaboratively as part of a team.