In addition to creating and maintaining an optimal pipeline architecture, key responsibilities for a Senior Data Engineer include:
Developing high-quality, high-performance, scalable applications using best practices and the latest generation of technologies
Building required infrastructure for optimal extraction, transformation and loading of data from various data sources using AWS and SQL technologies
Assembling large, complex sets of data that meet non-functional and functional business requirements.
Identifying, designing, and implementing internal process improvements including re-designing infrastructure for greater scalability, optimizing data delivery, and automating manual processes
Building analytical tools/dashboards to utilize the data pipeline, providing actionable insight into key business performance metrics including operational efficiency
Creating and supporting internal APIs and products that enable workflows.
Working with stakeholders including data, design, product and executive teams and assisting them with data-related technical issues
Following Agile best practices effectively and actively participate in all scrum ceremonies
Being part of a culture of continuous learning and improvement of both the team and codebase
Responding to and resolving production issues
Desired Skills
Ability to build and optimize data sets, data pipelines and architectures
Ability to perform root cause analysis on external and internal processes and data to identify opportunities for improvement
Excellent analytic skills associated with working on unstructured datasets
Ability to build processes that support data transformation, workload management, data structures, dependency and metadata
Experience with auto scaling, performance testing, and capacity planning
Experience owning infrastructure in production, as well as designing and creating build/deploy & monitoring systems
Exceptional analytical and problem-solving skills