1,004 Hadoop jobs in Singapore

Cloud Kinetics Technology Solutions Private Ltd | Full time

Cloudera Hadoop Administrator

Singapore, Singapore | Posted on 10/08/2024

Responsibilities

• Administering large scale Cloudera Hadoop environments build and support including design, cluster set up, performance tuning and monitoring in an enterprise environment.
• Installing, administering, and supporting Linux operating systems and hardware.
• System administration and performance tuning.
• Setup, configuration and management of security.
• Manage and monitor platform infrastructure.
• Automate cluster node provisioning and repetitive tasks.
• Responsible for cluster availability.
• Support development and production deployments.
• Support disaster recovery and business continuity practice.

Requirements

Required Skills:

• 3+ years in Cloudera Hadoop Administration.
• 3+ years of experience in Hadoop and related technology stack, Elastic search is mandatory.
• Strong understanding of Hadoop design principles, cluster connectivity, security and the factors that affect distributed system performance.
• Strong software development or scripting skills.
• Strong knowledge and experience in supporting Linux environments.
• Experience maintaining, troubleshooting and setup large clusters.
• Experience supporting systems with 24X7 availability and monitoring.
• Strong knowledge of current hardware systems commonly used in production environments.
• Ability to proactively identify, troubleshoot and resolve live systems issues.

Duration: 1 year renewable contract

Working Location: Changi Business Park / Marina Bay Financial Centre

Job Overview

• We are looking for an experienced and motivated Data Engineer with expertise in Scala, SQL, and Hadoop to join our growing team. In this role, you will be responsible for designing, developing, and maintaining scalable data pipelines and systems that handle large datasets. You will collaborate with cross-functional teams to deliver high-quality solutions for data storage, transformation, and analytics.

Key Responsibilities:

• Design, build, and optimize scalable data pipelines using Scala and Hadoop frameworks.
• Implement ETL processes to ingest, transform, and store data from various sources.
• Write and optimize complex SQL queries for efficient data retrieval and transformation.
• Troubleshoot and resolve performance issues in queries and data workflows.
• Manage and maintain Hadoop-based data infrastructure, including HDFS, Hive, and related components.
• Monitor system performance and optimize resource utilization in a distributed environment.
• Work closely with data analysts, data scientists, and business stakeholders to understand requirements and translate them into technical solutions.
• Provide technical guidance and promote best practices for data engineering within the team.
• Develop scripts and tools to automate repetitive tasks and improve system monitoring.
• Ensure data quality and consistency through validation and monitoring mechanisms.
• Document technical solutions, workflows, and processes to enable knowledge sharing and reproducibility.
• Ensure adherence to data governance and security policies.

Requirements:

• 4-6 years of experience in data engineering, big data technologies, or a similar role.
• Expertise in Scala programming and functional programming concepts.
• Strong SQL skills, with a focus on query optimization and database design.
• Hands-on experience with Hadoop ecosystem tools (HDFS, Hive, YARN, MapReduce).
• Familiarity with additional big data tools like Spark, Kafka, or Flink is a plus.
• Strong analytical skills, with the ability to troubleshoot and resolve complex data-related issues.
• Excellent communication skills to collaborate effectively with both technical and non-technical stakeholders.

Junior Data Engineer (Hadoop & Spark) - Contract- Changi

Employer: NTT DATA Singapore

Work Location: Changi , Singapore (Onsite, within a leading

Junior Data Engineer (Hadoop & Spark) - Contract- Changi

Employer: NTT DATA Singapore

Work Location: Changi , Singapore (Onsite, within a leading

Junior Data Engineer (Hadoop & Spark) - Contract- Changi

Employer: NTT DATA Singapore

Work Location: Changi , Singapore (Onsite, within a leading

• Understands the problem statement and requirement of the current processes from the relevant stakeholders
• Participates in the data collection, cleansing and matching and understand of the current set of data which required to be analysed
• Analyses the data both structured and unstructured set across different categories based on relevant stakeholders
• Develops automation scripts to perform the task efficiently for current systems which is operationalised
• To use AI tools to analyse the integrated data set for insights
• Develops AI/ML based solutions based on various IOT inputs including model trainings and detection for visual analytics
• Monitors the performance and ensures the analytics solutions deployed are executed optimally and support in the troubleshooting and refinement process
  
  

• Power BI, Power Automate, MS Office Suites, Python, Unity, Data Fabric will be good skills to have
• Good Communication and Interpersonal Skills, Problem-Solving Abilities, Self-Motivated
• The candidate will be assigned mentor to learn these skills during the internship as well