Job Title: Data Engineer for Large-Scale Data Systems
About the Role
We are seeking an experienced Data Engineer to design, build, and maintain large-scale data systems. As a key member of our team, you will work closely with data architects, data scientists, and other stakeholders to ensure that our data systems meet the needs of our business.
Key Responsibilities:
* Data System Design and Development: Design, build, and maintain large-scale data systems, including data pipelines, data warehouses, and data lakes.
* Database Administration: Administer databases using tools such as Amazon Redshift, Google BigQuery, and Snowflake.
* Automated Testing Solutions: Design and implement scalable automated testing solutions using Ruby/Selenium-based frameworks.
* Data Pipeline Development: Develop and maintain data pipelines using tools such as Apache Beam, Apache Spark, and AWS Glue.
* Data Lake Development: Develop and maintain data lakes using tools such as Apache Hadoop, Apache Spark, and Amazon S3.
Requirements:
* 5+ years of experience in data engineering or a related field
* 2-4 years of experience in Ruby products, including Ruby on Rails framework
* 5+ years of experience with programming languages such as Python, Java, and Scala
* 3+ years of experience with data modeling and data architecture
* 3+ years of experience with data engineering tools such as Apache Beam, Apache Spark, AWS Glue, Amazon Redshift, Google BigQuery, and Snowflake
Preferred Qualifications:
* Experience with machine learning and data science
* Experience with cloud-based data platforms such as AWS, GCP, or Azure
* Experience with containerization using Docker and Kubernetes
* Experience with agile development methodologies such as Scrum or Kanban