Talent.com
Ta oferta pracy nie jest dostępna w Twoim kraju.
Big Data Engineer

Big Data Engineer

RTB HouseWarsaw, Masovian Voivodeship, Poland
Ponad 30 dni temu
Opis pracy

We are looking for experienced developers who will help to build and expand our data processing platform.

Your tasks :

  • Development and maintenance of distributed systems processing large amounts of data (most real-time) for the needs of our RTB platform
  • Optimization of the developed software in terms of efficiency and resource consumption
  • Ensuring the reliability and scalability of the solutions built
  • Creating performance and correctness tests for new system components
  • Analysis of new technologies in terms of their applicability in production conditions
  • Development of tools for monitoring and analyzing the operation of the production system
  • Continuous optimization of existing tools and processes

Selected technologies used :

  • Java, Python
  • Hadoop, Kafka
  • Kafka Streams, Flume, Logstash
  • Docker, Jenkins, Graphite
  • Aerospike, PostgreSQL
  • Google Big Query, Elastic

    • Selected issues that we have dealt with recently :

  • Replacement of the framework in the data processing component (transition from Storm to Kafka Streams)
  • Creating a data stream merger based on the Kafka Client API
  • Creating a user profile synchronizer between DC's based on Kafka Streams
  • Creating a component that calculates aggregates based on the Kafka Client API and Bloom filters
  • Implementation of Logstash for loading and Elastic for querying indexed data (transition from Flume + Solr)
  • Creating end-to-end monitoring of data correctness and delay
  • Replacement of the data streaming component to BigQuery and HDFS (from Flume to a proprietary solution based on Kafka Client API)
  • Continuous system maintenance, detection and resolution of performance problems, as well as scaling due to the growing amount of data

    • Our expectations :

  • Proficiency in programming
  • Excellent understanding of how complex IT systems work (from the hardware level, through software, to algorithmics)
  • Good knowledge of basic methods of creating concurrent programs and distributed systems (from thread level to continental level)
  • Practical ability to observe, monitor and analyze the operation of production systems (and draw valuable conclusions from it)
  • The ability to critically analyze the solutions created in terms of efficiency (from estimating the theoretical performance of the designed systems to detecting and removing actual performance problems in production)
  • Readiness to work in the DevOps model

    • Additional advantages will be :

  • Experience in creating distributed systems
  • Good knowledge of selected Big Data technologies such as Hadoop, Kafka, Storm, Spark or Flink
  • Knowledge of application profiling methods and tools (preferably Java, both from the JVM and Linux level)

    • We offer :

  • Attractive salary
  • Work in a team of enthusiasts who are willing to share their knowledge and experience
  • Extremely flexible cooperation conditions - we do not have core hours, we do not have holiday limits, you can work fully remotely
  • Access to the latest technologies and the possibility of real use of them in a large-scale and highly dynamic project

    • Do you have questions about the project, team, style of work? Visit our tech blog : http : / / techblog.rtbhouse.com / jobs /

    Utwórz powiadomienie o ofertach pracy dla tego wyszukiwania

    Data Engineer • Warsaw, Masovian Voivodeship, Poland