Responsibilities:
As a Big Data Engineer, you will develop, maintain, evaluate, and test big data solutions. You will be involved in data engineering activities such as the following:
• Creating pipelines or workflows for the source-to-target data journey
• Designing data solutions by using big-data-based technologies along with Hadoop, MongoDB, Casandra, and Azure HDInsight for Cloudera-based data lake by using Scala Programming
• Monitoring data pipelines and DataOps
• Ingesting data from files, streams, and databases
• Processing the data with Hadoop, Scala, SQL Database, Spark, ML, and IoT devices, among others
• Developing programs in Scala and Python as a part of data cleaning and processing
• Designing and developing distributed, high volume, high-velocity multi-threaded event processing systems
• Developing efficient software code for multiple use cases leveraging Python and big data technologies for various use cases built on the platform
• Providing high operational excellence guaranteeing high availability and platform stability
• Implementing scalable solutions to meet the ever-increasing data volumes, using big data/cloud technologies such as PySpark, Kafka, cloud computing, and others