Data Engineer

 

Roles and responsibilities:

·Implement scalable and sustainable data engineering
solutions using tools such as Databricks, Snowflake,
Teradata, Apache Spark and Python. The data pipelines
must be created, maintained and optimized as workloads
move from development to production for specific use
cases.
·Own end-to-end development, including coding, testing,
debugging and deployment.
·Drive automation through effective use of modern tools,
techniques and architectures to completely automate the
repeatable and tedious data preparation and integration
tasks to improve productivity.
·Map data between source systems, data warehouses, and
data marts.
·Train counterparts in the data pipelining and preparation
techniques, which make it easier for them to integrate and
consume the data they need for their own use cases.. 
·Interface with other technology teams to extract,
transform, and load data from a wide variety of data
sources.
·Promote the available data and analytics capabilities and
expertise to business unit leaders and educate them in
leveraging these capabilities in achieving their business
goals.
·Translate SQL queries into python code running on a
distributed system
·Develop libraries to re-use code
·Excellent verbal and communication skills.
Requisites
Core Competencies :
·Must have experience working with Spark/Hadoop
development (Spark preferred)
·Must have experience working in Python with NumPy and
Pandas (PySpark preferred)
·Expertise in SQL
·Database experience with proven ability to write complex

queries on large datasets
·Expert ability to build data pipelines and wrangle data
·Hands-on experience in Data warehousing tools.
·Knowledge of distributed systems such as Hadoop, Hive,
Spark, Kafka
·Proficient in a source code control system such as GIT

Technical Skills :

·Robust experience distributed systems such as Hadoop,
Hive, Spark, Kafka and DATABRICKS
·Robust SQL, PLSQL and Scripting skills
·Spark Streaming
·Languages: Java and Python
·Databases: Microsoft SQL & MySQL
·Datawarehouse: Teradata, snowflake
·Operating Systems: Windows & Linux
·Other Tools: Putty & WinSCP
·AWS
oCompute : EC2, Lambda
oStorage : S3
oAnalytics : Athena, Glue--Management Tools :
CloudWatch, Management Console, Command Line
Interface

Good to have:

·Experience with big data tools: Hadoop, Spark, Kafka, etc.
·Experience with relational SQL and NoSQL databases,
including Postgres and Cassandra.
·Experience with data pipeline and workflow management
tools: Rundeck, Airflow, etc.
·Experience with AWS cloud services: EC2, EMR, RDS,
Redshift
·Experience with stream-processing systems: Storm, Spark-
Streaming, etc.
·Experience with object-oriented/object function scripting