Pyramid Systems, Inc.

Sr. Data Engineer

Job Locations US-DC
Posted Date 1 month ago(3/6/2024 3:57 PM)
Job ID
2024-2003
# of Openings
1

Overview

Pyramid Systems is looking for a Data Engineer (Senior) who is passionate about bringing creative architect solutions to end customers.

Responsibilities

  • You'll work with AI team members to operationalize data pipelines and ML tasks
  • You'll provide day-to-day support of deploying Python-native ML pipelines
  • You'll perform data engineering tasks to enable AI/ML capabilities
  • You'll present results to a diverse audience in presentation or report form
  • You'll support architectural leadership, technical support, and advisement services to ensure identity management system technologies are integrated and meeting the appropriate security requirements
  • You’ll support leadership who engage with senior level executives at a public facing Federal agency and provide subject matter expertise in security architecture and other key domain areas

Qualifications

  • You have the ability to obtain and maintain DHS Suitability
  • You are a US Citizen
  • Plan, create, and maintain data architectures, ensuring alignment with business requirements.
  • Obtain data, formulate dataset processes, and store optimized data.
  • Experiance in Databricks, PySpark and SQL
  • Identify problems and inefficiencies and apply solutions.
  • Determine tasks where manual participation can be eliminated with automation.
  • "Identify and optimize data bottlenecks, leveraging automation where possible
  • Create and manage data lifecycle policies (retention, backups/restore, etc.)"
  • Create, maintain and manage ETL/ELT pipelines & data transformations
  • Create, maintain and manage data quality gates (Great Expectations) to ensure high data quality
  • Support the AI/ML teams with optimizing feature engineering code
  • Lots and lots of Spark updates
  • Create, maintain and manage Spark Structured Streaming jobs, including using the newer Delta Live Tables and/or DBT
  • Research existing data in the data lake to determine best sources for data
  • Create, manage and maintain ksqlDB and Kafka Streams queries/code
  • Maintain and update Python-based data processing scripts executed on AW Lambdas
  • Unit tests for all our Spark, Python data processing, and Lambda codes
  • Maintain our PCIS Reporting Database data lake with optimizations and maintenance (perf tuning, etc.)

Education 

  • Bachelor's degree in Computer Science or related discipline

Options

Sorry the Share function is not working properly at this moment. Please refresh the page and try again later.
Share on your newsfeed