Job Description
As a data engineering professional, you will be responsible for designing and implementing a serverless architecture to efficiently handle large volumes of data. You will develop data ingestion pipelines and integration processes to ensure the smooth transfer of data from various sources.
You will implement data transformation and enrichment processes using AWS Lambda, Glue, or similar technologies to ensure data quality and consistency. You will collaborate with data scientists and analysts to understand their data requirements and design appropriate models and schemas.
You will optimize storage and retrieval mechanisms, leveraging AWS services such as S3, Athena, Redshift, or DynamoDB, to provide high-performance access to the data. You will monitor and troubleshoot the infrastructure, identifying performance bottlenecks, data processing errors, and other issues.
Key skills include:
* AWS technologies and serverless architectures
* Strong programming skills in languages like Python, Java, or Scala
* Experience with data integration tools, such as AWS Glue or Apache Spark
* Data modeling techniques and data warehousing concepts
* Data visualization tools, such as Tableau or Power BI