About the Role
\
We are seeking a skilled Data Engineer to join our collaborative environment and help build and maintain our data infrastructure. As a key member of our team, you will be responsible for designing, building, and maintaining large-scale data systems, including data pipelines, data warehouses, and data lakes.
\
Key Responsibilities:
\
\
1. Design and implement scalable automated testing solutions using Ruby/Selenium-based frameworks.
\
2. Develop and maintain data pipelines using tools such as Apache Beam, Apache Spark, and AWS Glue.
\
3. Develop and maintain data lakes using tools such as Apache Hadoop, Apache Spark, and Amazon S3.
\
4. Work with data architects to design and implement data models and data architectures.
\
5. Collaborate with data scientists to develop and deploy machine learning models and data products.
\
6. Ensure data quality and integrity by developing and implementing data validation and data cleansing processes.
\
7. Collaborate with other teams to ensure that data systems meet the business's needs.
\
8. Stay up-to-date with new technologies and trends in data engineering and make recommendations for adoption.
\
9. Candidates should have strong experience with data warehousing and data lakes.
\
10. Candidates should have strong experience with data validation and data cleansing.
\
11. Candidates should have strong collaboration and communication skills.
\