About the Role:
We are seeking a skilled Data Engineer to join our team and contribute to the design, development, and maintenance of large-scale data systems. As a key member of our data infrastructure team, you will work closely with data architects, data scientists, and other stakeholders to ensure that our data systems meet the needs of our business.
Key Responsibilities:
1. Data System Development: Design, build, and maintain large-scale data systems, including data pipelines, data warehouses, and data lakes.
2. Data Warehousing: Design and implement data warehouses using tools such as Amazon Redshift, Google BigQuery, and Snowflake.
3. Automated Testing: Design and implement scalable automated testing solutions using Ruby/Selenium-based frameworks.
4. Data Pipelines: Develop and maintain data pipelines using tools such as Apache Beam, Apache Spark, and AWS Glue.
5. Data Lakes: Develop and maintain data lakes using tools such as Apache Hadoop, Apache Spark, and Amazon S3.
6. Data Modeling: Work with data architects to design and implement data models and data architectures.
7. Machine Learning: Collaborate with data scientists to develop and deploy machine learning models and data products.
8. Data Quality: Ensure data quality and integrity by developing and implementing data validation and data cleansing processes.
9. Collaboration: Collaborate with other teams to ensure that data systems meet the business's needs.
10. Professional Development: Stay up-to-date with new technologies and trends in data engineering and make recommendations for adoption.