Our legacy data acquisition framework consists of a set of monolithic shell and python based utilities. Adding new data sources is slow, testing is complicated, error handling is difficult. We would like to evaluate Open Source data acquisition projects and rework our infrastructure either by adopting and tailoring one of the Open Source solutions or implementing our own based on the lessons learned. We would like to implement hooks for error handling, re-runs and targeted alerting. Our ideal framework is python based, object oriented, secure, easy to extend and maintain. We would like to achieve high unit test coverage and easy integration testing.
Responsibilities
Understand legacy architecture, tooling, functions and requirements
Evaluate Open Source data acquisition framework and make recommendation for adaptation
Collaborate on requirements refinement and estimation
Design new object structure and develop efficient and scalable components
Implement error handling, hooks for reruns and alerting
Participate in evolving our engineering, coding and testing standards
Role requirements
Ideal candidate has strong background in Python Object Oriented Programming and Test Driven Development with a least a few years in leading role.
Good understanding of clean coding principles, object oriented design and patterns
Experience in code refactoring
Experience creating high quality documentationdescribing requirements and implementation detail
Ability to write scalable, reusable, testable, efficient code
Good understanding of Unix shell scripts, networking protocols, REST
Good understanding of SQL
Experience in processing various data formats, good understanding of processing XML, JSON in Python