DAI
Senior Specialist, Data Engineer
DAI
1973 views
Posted date 5th October, 2022 Last date to apply 16th October, 2022
Country Pakistan Locations Remote US
Category Engineering
Type Full Time Positions 1
Status Closed

DAI is a global development company with corporate offices in the United States, the United Kingdom, EU, Nigeria, Pakistan, and Palestine and project operations worldwide. We tackle fundamental social and economic development problems caused by inefficient markets, ineffective governance, and instability. DAI works on the frontlines of global development. Transforming ideas into action—action into impact. We are committed to shaping a more livable world.

DAI and its employees are committed to confronting racism and holding ourselves accountable for positive change within the company and in the communities, cultures, and countries in which we live and work.  DAI is committed to attracting and retaining the best employees from all races and backgrounds in our continued effort to become a better development partner.

DAI upholds the highest ethical standards. We are committed to the prevention of sexual exploitation, abuse, and harassment as well as other ethical breaches. All of our positions are therefore subject to stringent vetting and reference checks.

DAI is an equal opportunity/affirmative action employer with a commitment to diversity. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.

Summary of the Position

Many of DAI’s legacy systems, including its project information management system used around the globe, are undergoing a strategic transition to new platforms that will support DAI’s strategy to become a data-enabled, cloud-first organization. This reflects the continuing technological change and our clients’ increased data collection, reporting, and integration demands. The successful candidate will design and implement Azure data storage solutions, data processing, security, integration solutions, cloud analytic solutions, data migration, internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.

DAI’s Current Technology stack includes, but is not limited to, Oracle ERP, Azure Cloud, MS 365/.Net (including Dynamics, Power App, and Power BI), SharePoint, Lotus Notes, and a variety of EUCs. This position reports to the Senior Manager, Data Management. 

Responsibilities

  • Design & Implement Azure data storage solutions, data processing, security, integration solutions, cloud analytic solutions, data migration, internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
  • Create and maintain optimal data pipeline architecture, assemble large, complex data sets that meet functional / non-functional business requirements.
  • Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL and ‘big data’ technologies.
  • Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency, and other key business performance metrics.
  • Work with stakeholders on Product, Data and Design teams to assist with data-related technical issues and support their data infrastructure needs.
  • Create data tools for analytics and data scientist team members that assist them in building and optimizing the analytics and data accuracy.
  • Work with data and analytics experts to strive for greater functionality in our data systems.
  • Work on collecting, storing, processing, and analyzing huge sets of data. The primary focus will be on choosing optimal solutions to use for these purposes, then maintaining, implementing, and monitoring them
  • Work with relational and non-relational databases, query authoring (SQL) as well as working familiarity with a variety of databases and data file formats for used for ETL/ELT processes.
  • Build processes supporting data transformation, data structures, metadata, dependency, and workload management
  • Selecting and integrating any Big Data tools and frameworks required to provide requested capabilities
  • Performing Regression to verify and uncover new errors in legacy code after changes have been made.
  • Contributes ideas on product and process improvements. Participates in design review sessions.
  • Implementing ETL process, if importing data from existing data sources as relevant and Performance tuning to reduce the data processing times.
  • Monitoring performance and advising any necessary infrastructure changes
  • Organize and attend daily standup meetings with other data engineers and members of the data team
  • Design and implement the data management, monitoring, security, and privacy of data using the full stack of Azure data services to satisfy business needs.
  • There will be frequent interaction with project management both locally and overseas and the need to work with customers/team members across different time zones.

Additional responsibilities as deemed necessary.

Qualifications & Skills

Minimum Requirements

  • A college degree, plus 5 years of relevant experience. Equivalent combinations of education and experience will be considered.
  • Demonstrated experience utilizing data modeling, ETL/ETL, Azure synapse analytics, AAS, Azure SQL, T-SQL, Python, Scala, R, Azure Databricks, Azure data factory, Azure SQL, Elastic pools, Azure Data Lake/Blob Storage, SSIS, SSAS, SSDT, Stream Analytics, Power BI and other technologies like IoTHub, Redis as required.
  • Demonstrated data ingestion experience from various data sources, databases, APIs, files etc.
  • Proven experience utilizing distributed data/computing tools: Map/Reduce, Hadoop, Hive, Spark, Gurobi, MySQL, etc.
  • Demonstrated experience implementing AAS to provide centralized business rules, KPI's, Hierarchies, and data access.
  • Possess understanding of of DevOps practices, Repos, Build and release CICD pipelines etc.
  • Proven ability to apply critical thinking to develop and solve business challenges.
  • Demonstrated experience in Azure services like Log Analytics, Network Watcher, Application Insights Knowledge of enterprise design patterns and cloud design patterns.
  • Demonstrated ability to read and understand system requirements, formulate a software design approach based on those requirements.
  • Demonstrated experience in developing and testing software that is aligned with defined coding standards.
  • Proven background in developing products and troubleshooting and understanding of SDLC and bug tracking process.
  • Demonstrated experience in building stream-processing systems, data ingestion using solutions and knowledge of ETL techniques and frameworks.
  • Demonstrated ability to translate complex functional and technical requirements into detailed design.
  • Proven experience configuring, management and Deployment of Big Data applications
  • Demonstrated ability to design, build, install, configure and support Azure components in addition to experience in Data Lake development and implementation
  • Demonstrated experience using object-oriented programming languages like C#, PowerShell, Python etc.

Equal Employment Opportunity

DAI and its employees are committed to confronting racism and holding ourselves accountable for positive change within the company and in the communities, cultures, and countries in which we live and work.  DAI is committed to attracting and retaining the best employees from all races and backgrounds in our continued effort to become a better development partner.

DAI upholds the highest ethical standards. We are committed to the prevention of sexual exploitation, abuse, and harassment as well as other ethical breaches. All of our positions are therefore subject to stringent vetting and reference checks.

DAI is an equal opportunity/affirmative action employer with a commitment to diversity. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.

DAI requires COVID-19 vaccinations subject to accommodation required by law.

Related
Sub Engineer

Responsible to:             Progra....