493 Senior Data Engineer jobs in Singapore

Direct message the job poster from Baidu, Inc.

-Build the company's big data warehouse system, including batch and stream data flow construction;

-In-depth understanding of business systems, understanding of project customer needs, design and implement big data systems that meet user needs, and ensure smooth project acceptance

- Responsible for data integration and ETL architecture design and development

- Research cutting-edge big data technologies, optimize big data platforms continuously

Job requirements:

-Bachelor degree or above in computer, database, machine learning and other related majors, more than 3 years of data development work;

-Have a keen understanding of business data, and can analyze business data quickly;

-Proficient in SQL language, familiar with MySQL, very familiar with Shell, Java (Scala), Python;

-Familiar with common ETL technologies and principles; proficient in data warehouse database design specifications and practical operations; rich experience in spark, MR task tuning

-Familiar with Hadoop, Hive, HBase, Spark, flink, kafka and other big data components ;

-Proficient in the use of mainstream analytical data systems such as clickhouse、doris、 TIDB, have tuning experience are preferredBig Data Development Engineer

• Seniority level Mid-Senior level

• Employment type Full-time

• Job function Information Technology and Engineering
• Industries Software Development, Technology, Information and Media, and Information Services

Referrals increase your chances of interviewing at Baidu, Inc. by 2x

Get notified about new Big Data Developer jobs in Singapore, Singapore .

South East Community Development Council, Singapore 1 day ago

We’re unlocking community knowledge in a new way. Experts add insights directly into each article, started with the help of AI.

Design and implement scalable data pipelines to collect, process, and store large volumes of structured and unstructured data.

Ensure data quality, integrity, and consistency across various data sources.

Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.

Integrate data from multiple sources, including databases, APIs, and third-party data providers.

Build and maintain big data infrastructure using technologies

Optimize data storage and retrieval processes to improve performance and reduce costs.

Proven experience as a Big Data Engineer or similar role.
Experience with big data technologies such as Hadoop, Spark, Kafka, Hive, HBase, etc

Proficiency in programming languages such as Python, Java, or Scala.
Strong knowledge of SQL and NoSQL databases.
Knowledge of machine learning and data analytics.

About TikTok

TikTok is the leading destination for short-form mobile video. At TikTok, our mission is to inspire creativity and bring joy. TikTok's global headquarters are in Los Angeles and Singapore, and we also have offices in New York City, London, Dublin, Paris, Berlin, Dubai, Jakarta, Seoul, and Tokyo.

Why Join Us

Inspiring creativity is at the core of TikTok's mission. Our innovative product is built to help people authentically express themselves, discover and connect – and our global, diverse teams make that possible. Together, we create value for our communities, inspire creativity and bring joy - a mission we work towards every day.

We strive to do great things with great people. We lead with curiosity, humility, and a desire to make impact in a rapidly growing tech company. Every challenge is an opportunity to learn and innovate as one team. We're resilient and embrace challenges as they come. By constantly iterating and fostering an "Always Day 1" mindset, we achieve meaningful breakthroughs for ourselves, our company, and our users. When we create and grow together, the possibilities are limitless. Join us.

Diversity & Inclusion

TikTok is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At TikTok, our mission is to inspire creativity and bring joy. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too.

Job highlights

Positive team atmosphere, Career growth opportunity, Paid leave, 100+ mil users, Meals provided

Responsibilities

Team Introduction

The mission of the Data Platform Singapore Business Partnering (DPSG BP) team is to empower the TikTok Business with data. Our goal is to build a Data Warehouse that can cater to batch and streaming data, Data Products that provide useful information to build efficient data metrics & dashboards which will be used to make smarter business decisions to support business growth. If you're looking for a challenging ground to push your limits, this is the team for you!

Responsibilities:

- Translate business requirements & end to end designs into technical implementations and responsible for building batch and real-time data warehouse.

- Manage data modeling design, writing, and optimizing ETL jobs.

- Collaborate with the business team to building data metrics based on data warehouse.

- Responsible for building and maintaining data products.

- Involvement in rollouts, upgrades, implementation, and release of data system changes as required for streamlining of internal practices.

- Develop and implement techniques and analytics applications to transform raw data into meaningful information using data-oriented programming languages and visualisation software.

- Apply data mining, data modelling, natural language processing, and machine learning to extract and analyse information from large structured and unstructured datasets.

- Visualise, interpret, and report data findings and may create dynamic data reports as well.

Qualifications

Minimum Qualifications:

- At least 5 years in software engineering and 2 years of relevant experience in data engineering.

- Proficient in creating and maintaining complex ETL pipelines end-to-end while maintaining high reliability and security.

- Familiar with data warehouse concept and have production experience in modeling design.

- Familiar with at least 1 distributed computing engine (e.g. Hive, Spark, Flink).

- Familiar with at least 1 NoSQL database is a plus (e.g. HBase).

Preferred Qualifications:

- Excellent interpersonal and communication skills with the ability to engage and manage internal and external stakeholders across all levels of seniority.

- Strong collaboration skills with the ability to build rapport across teams and stakeholders.

What's on offer:

· Location: Singapore

· Client: End client user environment

Job Summary:

We are seeking a highly skilled and motivated Lead Big Data Engineer to join our data team. The ideal candidate will play a key role in designing, developing, and maintaining scalable big data solutions while providing technical leadership. This role will also support strategic Data Governance initiatives, ensuring data integrity, privacy, and accessibility across the organization.

Key Responsibilities:

· Design, implement, and optimize robust data pipelines and ETL/ELT workflows using SQL and Python .

· Lead architecture discussions, including the creation and review of Entity Relationship Diagrams (ERDs) and overall system design.

· Collaborate closely with Data Engineers, Analysts, and cross-functional engineering teams to meet evolving data needs.

· Deploy and manage infrastructure using Terraform and other Infrastructure-as-Code (IaC) tools.

· Develop and maintain CI/CD pipelines for deploying data applications and services.

· Leverage strong experience in AWS services (e.g., S3, Glue, Lambda, RDS, Lake Formation) to support scalable and secure cloud-based data platforms.

· Handle both batch and real-time data processing effectively.

· Apply best practices in data modeling and support data privacy and data protection initiatives.

· Implement and manage data encryption and hashing techniques to secure sensitive information.

· Ensure adherence to software engineering best practices including version control, automated testing, and deployment standards.

· Lead performance tuning and troubleshooting for data applications and platforms.

Required Skills & Qualifications:

· Strong proficiency in SQL for data modeling, querying, and transformation.

· Advanced Python development skills with an emphasis on data engineering use cases.

· Hands-on experience with Terraform for cloud infrastructure provisioning.

· Proficiency with CI/CD tools, particularly GitHub Actions .

· Deep expertise in AWS cloud architecture and services.

· Demonstrated ability to create and evaluate ERDs and contribute to architectural decisions.

· Strong communication and leadership skills with experience mentoring engineering teams.

Preferred Skills:

· Experience with big data technologies such as Apache Spark , Hive , or Kafka .

· Familiarity with containerization tools (e.g., Docker) and orchestration platforms (e.g., Kubernetes).

· Solid understanding of data governance, data quality, and security frameworks

Design and implement scalable data pipelines to collect, process, and store large volumes of structured and unstructured data.

Ensure data quality, integrity, and consistency across various data sources.

Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.

Integrate data from multiple sources, including databases, APIs, and third-party data providers.

Build and maintain big data infrastructure using technologies

Optimize data storage and retrieval processes to improve performance and reduce costs.

Proven experience as a Big Data Engineer or similar role.
Experience with big data technologies such as Hadoop, Spark, Kafka, Hive, HBase, etc

Proficiency in programming languages such as Python, Java, or Scala.
Strong knowledge of SQL and NoSQL databases.
Knowledge of machine learning and data analytics.

• Experience : 10+ years
• (SQL Server / Oracle / DB2 / Netezza) – at-least good working knowledge in 2 of these DBApache Spark Streaming or Apache Flink
• Kafka
• NOSQL databases - Cosmos DB, Document DB
• Spark,Dataframe API
• Hive (HQL)
• Scripting language – Shell or bash
• CI CD
• Experience with at least one Cloud Infra provider (Azure/AWS)
  

Good to have Skills :

• Certifications related to Data and Analytics

About TikTok

TikTok is the leading destination for short-form mobile video. At TikTok, our mission is to inspire creativity and bring joy. TikTok's global headquarters are in Los Angeles and Singapore, and we also have offices in New York City, London, Dublin, Paris, Berlin, Dubai, Jakarta, Seoul, and Tokyo.

Why Join Us

Inspiring creativity is at the core of TikTok's mission. Our innovative product is built to help people authentically express themselves, discover and connect – and our global, diverse teams make that possible. Together, we create value for our communities, inspire creativity and bring joy - a mission we work towards every day.

We strive to do great things with great people. We lead with curiosity, humility, and a desire to make impact in a rapidly growing tech company. Every challenge is an opportunity to learn and innovate as one team. We're resilient and embrace challenges as they come. By constantly iterating and fostering an "Always Day 1" mindset, we achieve meaningful breakthroughs for ourselves, our company, and our users. When we create and grow together, the possibilities are limitless. Join us.

Diversity & Inclusion

TikTok is committed to creating an inclusive space where employees are valued for their skills, experiences, and unique perspectives. Our platform connects people from across the globe and so does our workplace. At TikTok, our mission is to inspire creativity and bring joy. To achieve that goal, we are committed to celebrating our diverse voices and to creating an environment that reflects the many communities we reach. We are passionate about this and hope you are too.

Job highlights

Positive team atmosphere, Career growth opportunity, Flat organization

Responsibilities

About the team

The mission of the Data Platform Singapore Business Partnering (DPSG BP) team is to empower the TikTok Business with data. Our goal is to build a Data Warehouse that can cater to batch and streaming data, Data Products that provide useful information to build efficient data metrics & dashboards which will be used to make smarter business decisions to support business growth. If you're looking for a challenging ground to push your limits, this is the team for you!

As a data engineer in the Data Platform team, you will have the opportunity to build, optimize and grow one of the largest data platforms in the world. You'll have the opportunity to gain hands-on experience on all kinds of systems in the data platform ecosystem. Your work will have a direct and huge impact on the company's core products as well as hundreds of millions of users.

Responsibilities:

- Design and build data transformations efficiently and reliably for different purposes (e.g. reporting, growth analysis, and multi-dimensional analysis).

- Design and implement reliable, scalable, robust and extensible big data systems that support core products and business.

- Establish solid design and best engineering practice for engineers as well as non-technical people.

- Develop and implement techniques and analytics applications to transform raw data into meaningful information using data-oriented programming languages and visualisation software.

- Apply data mining, data modelling, natural language processing, and machine learning to extract and analyse information from large structured and unstructured datasets.

- Visualise, interpret, and report data findings and may create dynamic data reports as well.

Qualifications

Minimum Qualifications:

- Bachelor or Masters degree in Computer Science or related technical field or equivalent practical experience.

- At least 3 years of experience in the Big Data technologies (Hadoop, M/R, Hive, Spark, Metastore, Presto, Flume, Kafka, ClickHouse, Flink etc.).

Preferred Qualifications:

- Experience with performing data analysis, data ingestion and data integration.

- Experience with schema design and data modeling.

- Experience with ETL (Extraction, Transformation & Loading) and architecting data systems.

- Experience in writing, analyzing and debugging SQL queries.

- Experience in data privacy and security related projects.

- Deep understanding of various Big Data technologies.

- Passionate and self-motivated about technologies in the Big Data area.

- Solid communication and collaboration skills.

We are seeking a highly skilled and experienced Big Data Engineer to join our team. The ideal candidate will be managing data engineering jobs in big data environment e.g., Cloudera Data Platform. The successful candidate will be responsible for designing, developing, and maintaining the data ingestion and processing jobs. Candidate will also be integrating data sets to provide seamless data access to users.

Responsibilities
• Analyse the Authority’s data needs and document the requirements.
• Refine data collection/consumption by migrating data collection to more efficient channels.
• Plan, design and implement data engineering jobs and reporting solutions to meet the analytical needs.
• Develop test plan and scripts for system testing, support user acceptance testing.
• Build reports and dashboards according to user requirements
• Work with the Authority’s technical teams to ensure smooth deployment and adoption of new solution.
• Ensure the smooth operations and service level of IT solutions.
• Support production issues

What we are looking for
• Good understanding and completion of projects using waterfall/Agile methodology.
• Strong SQL, data modelling and data analysis skills are a must.
• Hands-on experience in big data engineering jobs using Python, Pyspark, Linux, and ETL tools like Informatica.
• Hands-on experience in a reporting or visualization tool like SAP BO and Tableau is must.
• Hands-on experience in DevOps deployment and data virtualisation tools like Denodo will be an advantage.
• Track record in implementing systems using Hive, Impala and Cloudera Data Platform will be preferred.
• Good understanding of analytics and data warehouse i mplementations.
• Ability to troubleshoot complex issues ranging from system resource to application stack traces.
• Track record in implementing systems with high availability, high performance, high security hosted at various data centres or hybrid cloud environments will be an added advantage.
• Passion for automation, standardization, and best practices

CSIT develops products to advance the national security interests of Singapore. We use our products in a wide range of use-cases, including but not limited to Counter-terrorism and Computer Network Defence. We are looking for talented senior data engineers to lead and build mission-critical products that identify, analyse and investigate threats.

• As a senior data engineer in a team that is responsible for mission-critical data analytics systems, you will contribute to the development and maintenance of high-performance data pipelines and systems related to national security. You will be involved in complex technical projects to collect, parse and process large datasets with the aim to turn them into useful information and insights. Join us to seize the opportunity to use cutting-edge technologies to protect the nation.

• Design and build data pipelines that are scalable, robust and maintainable to ingest crucial data using cutting-edge platforms and technologies
• Work with product managers, engineering managers and key stakeholders to deliver impactful data products that meet our business needs
• Optimise performance of large-scale data pipelines for low-latency, high-throughput and reliability
• Ensure data integrity, quality and governance through robust design and operational monitoring
• Troubleshoot and resolve complex issues in critical data pipelines
• Implement best practices in data engineering, including testing, orchestration and monitoring of data pipelines
• Lead technical discussions with the team and key stakeholders to shape the future of our data products
• Mentor junior data engineers and contribute to building a strong data engineering culture within the team

• Bachelor's degree in computer science or equivalent
• 5+ years of experience in data engineering, with a strong focus on building large-scale data pipelines
• Proven experience in writing optimised ETL code to handle large datasets
• Proficient in programming languages such as Scala or Java
• Familiarity with the following technologies: Spark, Kafka, Hadoop, Elastic stack, Kudu
• Strong understanding of modern data architectures and data modelling paradigms
• Able to communicate technical details effectively and clearly
• Good interpersonal, analytical and problem-solving skills to get things done in a dynamic environment

• The work is purposeful and meaningful
• You will get to work with talented and driven people
• We work with modern technologies and tech stacks
• We have excellent engineering culture and work-life balance
• We aspire for engineering and operational excellence
• We empower to innovate

As CSIT is an agency under the Ministry of Defence (Singapore), only Singapore Citizens will be considered.