Data Management with Apache NiFi
2h 52mAdvanced2026-04-22
Authors
Janani Ravi
Certified Google Cloud Architect and Data Engineer
Course details
Explore the capabilities of Apache NiFi for dataflow automation and management. Learn how to set up and configure NiFi on Mac OS and Windows, and dive into creating, running, and monitoring dataflows. Find out how to split and merge FlowFiles, integrate NiFi with databases like PostgreSQL, and leverage cloud features with Amazon S3. Build your understanding of advanced NiFi concepts like concurrency, scheduling, and backpressure to optimize dataflows. Access the power of modular dataflows using process groups and master data manipulation across formats. Engage with real-world scenarios by integrating NiFi with HTTP endpoints and configuring alerts for monitoring. Ideal for data management professionals and anyone interested in streamlining data workflows, this course offers skill-building through real-world applications.
Concepts
Introduction
- Data management with Apache NiFi
- Prerequisites
Installing and Setting up Apache NiFi
- Apache NiFi core concepts
- Apache NiFi architecture
- Mac OS - Download Apache NiFi
- Windows - Download Apache NiFi
- Mac OS and Windows - Start the NiFi server
- Mac OS and Windows - Reset password and log into Apache NiFi
- Explore the NiFi UI
Building and Running Dataflows
- Add and configure processors
- Run our first dataflow
- Flow summaries and flow operations
Splitting and Merging FlowFiles
- Split flow files into multiple files
- Run and observe file splits
- Merge content into a single file
- Order files in a connection
- Download the flow definition
- The flow.json.fz file
Integrating NiFi with PostgreSQL
- Set up a PostgreSQL table
- Add get file and put database processors
- Configure the CSV reader controller service
- Configure the DBCP connection pool controller service
- Run and encounter dataflow errors
- Configure the ReplaceText processor
- Run the dataflow to insert data into PostgreSQL
- Configure the run schedule
- Configure task concurrency
Executing SQL Queries in NiFi
- Configure PutSQL for DML operations
- Execute SQL commands and convert write formats
- Convert between data formats
- Update attributes and writing files
- Run the flow end-to-end
- Modularize dataflows with process groups
- Run a dataflow with a process group
Integrating NiFi with Amazon S3
- Create a PostgreSQL table, Amazon S3 bucket, and an IAM user
- Read from files and store in a database
- Configure the PutS3Object processor
- Set up the AWS credential provider controller service
- Read from the file system, filter records, and write to Amazon S3
Using Advanced Features of NiFi Dataflows
- Configure backpressure
- Observe the effects of backpressure
- Configuring an InvokeHTTP processor
- Use a funnel in the dataflow
- Run a dataflow to read from a URL and write to files
Monitoring and Alerting in NiFi
- Set up monitoring for the dataflow
- Configure alerting for the dataflow
- Email alerts on dataflow success
- Email alerts on change in activity status
Conclusion
- Summary and next steps
Related courses
- Data Management with Apache NiFi (2023)
- Databricks Certified Data Engineer Associate Cert Prep
- Data Engineering Pipeline Management with Apache Airflow
- Apache Spark Essential Training: Big Data Engineering
- Databricks Certified Data Engineer Associate Cert Prep: 3 Incremental Data Processing
- Manage and Optimize Big Data with Apache Iceberg
- Learning Apache Airflow
- Tuning Kafka
Related learn paths
- Prepare for the Databricks Certified Data Engineer Associate Certification
- Advance Your Data Engineering Skills
- Explore a Career in Data Engineering
- Introduction to Fundamental Skills for Data Work: Data Strategy and Planning
- Master Data Engineering
- Advance Your Data Skills in Apache Spark
- Data Engineering Foundations Professional Certificate by Astronomer
- Moving from Data Scientist to Data Analyst