Data Engineer
Locations: Pune/Bengaluru
Experience: 03 - 05 years
Responsibilities:
As a Big Data Engineer, you will develop, maintain, evaluate, and test big data solutions. You will be involved in data engineering activities such as the following:
• Creating pipelines or workflows for the source-to-target data journey
• Designing data solutions by using big-data-based technologies along with Hadoop, MongoDB, Casandra, and Azure HDInsight for Cloudera-based data lake by using Scala Programming
• Monitoring data pipelines and DataOps
• Ingesting data from files, streams, and databases
• Processing the data with Hadoop, Scala, SQL Database, Spark, ML, and IoT devices, among others
• Developing programs in Scala and Python as a part of data cleaning and processing
• Designing and developing distributed, high volume, high-velocity multi-threaded event processing systems
• Developing efficient software code for multiple use cases leveraging Python and big data technologies for various use cases built on the platform
• Providing high operational excellence guaranteeing high availability and platform stability
• Implementing scalable solutions to meet the ever-increasing data volumes, using big data/cloud technologies such as PySpark, Kafka, cloud computing, and others
Requirements:
• Minimum 5+ years of experience in big data technologies
• High-level proficiency in any of the programming languages – Python, Scala, or Java
• Mid- to expert-level experience in programming capabilities in a large-scale enterprise
• In-depth experience in modern data platform components such as Hadoop, Hive, Pig, Spark, Python, Scala, Databricks, and Snowflake, among others
• Experience with Distributed Version Control environments such as GIT
• Familiarity with development tools – experience on either IntelliJ or Eclipse or VSCode IDE and Maven, among others
• Exposure to data visualization tools (Qlik Sense/Tableau/Metabase)
• Demonstrated experience in modern API platform design including how modern UIs are built consuming services and APIs
• Experience in data quality & governance, data security, data lake, data warehouse, lineage, Drools, metadata management, Apache workflow, and big data service in the cloud (AWS/Azure/GCP)
• Experience on Azure cloud including Data Factory, Databricks, Data Lake Storage
• Solid experience in all phases of software development life cycle – plan, design, develop, test, release, maintain, support, and decommission
