Pyspark Overview

PySpark is the Python API for Apache Spark, an open-source, distributed computing framework designed for big data processing. It allows Python developers to harness the power of Spark to perform large-scale data processing tasks across a cluster of machines, making it easier to work with big datasets efficiently.

If you want to work on large amount of dataset where pandas dataframe is not working fine so in such cases also pyspark is very useful.

PySpark supports all Spark's feature such as Spark SQL, Dataframes, Structured Streaming, Machine learning and Spark core.

SPARK SQL AND DATAFRAMES 

• Spark sql is apache spark module that is used for processing of structured data . It allows you to mix SQL queries with Spark programs , with PySpark dataframes you can efficiently read, write and analyze data using python and SQL. 

PANDAS API ON SPARK

• If you have large dataset , for that you are using pandas api, it usually used for small datasets and while working on large datasets the processing time will increase, in such cases we can divide the pandas api workload across multiple nodes.

STRUCTURED STREAMING

• Structured Streaming in PySpark is a scalable and fault-tolerant stream processing engine built on top of Spark SQL. It allows users to process real-time data streams as continuously updating tables in a declarative manner, similar to how batch jobs are handled in Spark.