Variant

Variant is a Databricks data engineering capability, introduced June 2024.

A column type for semi-structured data: one column holds arbitrary nested JSON-shaped data with no declared schema, created with parse_json() and queried by path, while shredding gives it query performance closer to real typed columns.

• Databricks recommends it as the replacement for keeping semi-structured data in JSON strings, but the swap can quietly change your answers: variant paths are matched case-sensitively, where JSON strings never cared. 🔗
• Its speed comes from quietly un-blobbing itself: shredding stores the fields that turn up often as real columns inside the Parquet files, so a query reads a column instead of picking through a binary blob. 🔗
• Shredding turns itself on for a new CREATE TABLE but pointedly not for CREATE OR REPLACE TABLE or ALTER TABLE - a deliberate refusal to change how anyone's existing tables and workloads already behave. 🔗

⚠ Limitations: A VARIANT column can't be a clustering key, partition, or Z-order key, and the type can't be used for comparisons, grouping, ordering, or set operations. Shredding adds write overhead, skips VARIANT nested inside arrays and maps, and never converts data already written (REORG TABLE ... APPLY (SHRED VARIANT) does that); reading or writing shredded tables needs Databricks Runtime 17.3 or above, with 18.1 or above recommended for variant statistics and data skipping. Migrating off JSON strings can break working queries, since paths become case-sensitive, the [*] array syntax is unsupported, and NULL is encoded differently. 🔗

Category

Data engineering

Introduced

June 2024🔗

Also known as

VARIANT, variant data type, variant shredding

Verified

2026-08-03