Turning PySpark into a Universal DataFrame API (github.com)

SQLFrame is a new compatibility layer that implements the PySpark DataFrame API so you can run PySpark-style transformation pipelines directly on SQL engines—no Spark cluster or Spark dependency required. By mapping DataFrame operations to generated SQL, SQLFrame lets engineers reuse existing PySpark code, run workloads natively in engines like BigQuery, DuckDB, Postgres, Snowflake, Databricks (and Spark itself), or run a "Standalone" session to produce SQL without any database connection. This reduces local development friction, leverages each engine’s query optimizer for production processing, and makes DataFrame logic easier to inspect, debug, and share as SQL. Technically, SQLFrame emits complex, engine-specific SQL (CTEs, window functions, casts/formatting) and can produce more human-readable SQL via df.sql(optimize=True). It defaults to a Spark dialect for input/output but supports configurable dialects so code feels native to the target engine. You can either activate SQLFrame to replace pyspark imports or instantiate engine-specific sessions; installs are available as pip extras (e.g., sqlframe[bigquery], [duckdb], [postgres], [snowflake], [databricks], [spark]) or via conda-forge. One engine (Redshift) is marked in-development with limited test coverage. SQLFrame also documents hooks to configure generated SQL and even to integrate OpenAI to refine SQL, making it a pragmatic tool for porting PySpark workflows onto SQL-first infrastructures.