DISCO CAST Data Engineer

DISCO CAST Data Engineer

Description

Amazon Music is awash in data! To help make sense of it all, the CAST team within DISCO (Data, Insights, Science & Optimization) team accelerates and facilitates content analytics and provides independence to generate valuable insights in a fast, agile, and accurate way. This domain provides analytical support for the below topics within Amazon Music: Programming / Label Relations / PR / Stations / Livesports / Originals / Case & CAM.

The CAST team enables repeatable, easy, in depth analysis of music customer behaviors. Our goal is to empower all teams at Amazon Music to make data driven decisions and effectively measure their results by providing high quality, high availability data, and democratized data access through self-service tools.

If you love the challenges that come with big data and devops, then this role is for you. We collect billions of events a day, manage petabyte scale data on Redshift and S3, and develop data pipelines using Spark/Scala EMR, SQL based ETL, Airflow and Java services.

We are looking for a talented, curious, enthusiastic, and detail-oriented Data Engineer, who knows how to take on big data challenges in an agile way. Duties include big data design and analysis, data modeling, and development, deployment, and operations of big data pipelines. You'll help build Amazon Music's most important data pipelines and data sets, and expand self-service data knowledge and capabilities through an Amazon Music data university.

The CAST team develops data specifically for a set of key business domains like personalization, playlist programming, merch, artists, and provides and protects a robust self-service core data experience for all internal customers. We deal in AWS technologies like Redshift, S3, EMR, EC2, DynamoDB, Kinesis Firehose, and Lambda.


Key job responsibilities
• Writing scripts, building microservices in AWS to increase team efficiency
• Building and managing ETL pipelines in AWS to ingest data from external vendors
• Assist the BIs on the team in managing our existing environment that consists of Redshift and SQL based pipelines. The activities around these systems will largely be well-defined via standard operation procedures (SOP) and typically involve approving data access requests, subscribing or adding new data to the environment, but there will be cases where creative problem-solving is required
• SQL data pipeline management (creating or updating existing pipelines) and maintenance tasks on the Redshift cluster
• Assist the team with the management of our next-generation AWS infrastructure. Tasks includes infrastructure monitoring via CloudWatch alarms, infrastructure maintenance through code changes or enhancements, and troubleshooting/root cause analysis infrastructure issues that arise, and in some cases this resource may also be asked to submit code changes based on infrastructure issues that arise.


About the team
Amazon Music is an immersive audio entertainment service that deepens connections between fans, artists, and creators.From personalized music playlists to exclusive podcasts, concert livestreams to artist merch, we are innovating at some of the most exciting intersections of music and culture.We offer experiences that serve all listeners with our different tiers of service: Prime members get access to all music in shuffle mode,and top ad-free podcasts, included with their membership; customers can upgrade to Music Unlimited for unlimited on-demand access to 100 million songs including millions in HD, Ultra HD, spatial audio and anyone can listen for free by downloading Amazon Music app or via Alexa-enabled devices.Join us for opportunity to influence how Amazon Music engages fans, artists, and creators on a global scale.

Basic Qualifications

- 2+ years of data engineering experience
- Experience with data modeling, warehousing and building ETL pipelines
- Experience with one or more query language (e.g., SQL, PL/SQL, DDL, MDX, HiveQL, SparkSQL, Scala)
- Experience with one or more scripting language (e.g., Python, KornShell)
- Experience in troubleshooting the issues related to data and infrastructure issues.
- A passion for problem solving
- Excessive curiosity
- Exceptional ability to learn quickly and independently

Preferred Qualifications

- Experience in Linux/Unix scripting a big plus (e.g., bash scripts)
- Experience writing CLI tools/utilities in any language
- Experience with Typescript/Javascript and Java
- Experience with tools such as DBT, Stitch, Glue, Airflow
- Knowledge of distributed systems as it pertains to data storage and computing
- Experience in administering reporting/analytics platforms