Pyramid Systems, Inc.

Sr. Data Engineer

Job Locations US-DC
Posted Date 1 month ago(3/6/2024 3:57 PM)
Job ID
# of Openings


Pyramid Systems is looking for a Data Engineer (Senior) who is passionate about bringing creative architect solutions to end customers.


  • You'll work with AI team members to operationalize data pipelines and ML tasks
  • You'll provide day-to-day support of deploying Python-native ML pipelines
  • You'll perform data engineering tasks to enable AI/ML capabilities
  • You'll present results to a diverse audience in presentation or report form
  • You'll support architectural leadership, technical support, and advisement services to ensure identity management system technologies are integrated and meeting the appropriate security requirements
  • You’ll support leadership who engage with senior level executives at a public facing Federal agency and provide subject matter expertise in security architecture and other key domain areas


  • You have the ability to obtain and maintain DHS Suitability
  • You are a US Citizen
  • Plan, create, and maintain data architectures, ensuring alignment with business requirements.
  • Obtain data, formulate dataset processes, and store optimized data.
  • Experiance in Databricks, PySpark and SQL
  • Identify problems and inefficiencies and apply solutions.
  • Determine tasks where manual participation can be eliminated with automation.
  • "Identify and optimize data bottlenecks, leveraging automation where possible
  • Create and manage data lifecycle policies (retention, backups/restore, etc.)"
  • Create, maintain and manage ETL/ELT pipelines & data transformations
  • Create, maintain and manage data quality gates (Great Expectations) to ensure high data quality
  • Support the AI/ML teams with optimizing feature engineering code
  • Lots and lots of Spark updates
  • Create, maintain and manage Spark Structured Streaming jobs, including using the newer Delta Live Tables and/or DBT
  • Research existing data in the data lake to determine best sources for data
  • Create, manage and maintain ksqlDB and Kafka Streams queries/code
  • Maintain and update Python-based data processing scripts executed on AW Lambdas
  • Unit tests for all our Spark, Python data processing, and Lambda codes
  • Maintain our PCIS Reporting Database data lake with optimizations and maintenance (perf tuning, etc.)


  • Bachelor's degree in Computer Science or related discipline


Sorry the Share function is not working properly at this moment. Please refresh the page and try again later.
Share on your newsfeed