Most questions come down to precise method behavior: argument order, defaults, and which operations shuffle. Each table below pairs a concept with the detail that decides the answer. Read the left column, answer from memory, then check the right one.
The exam: 45 scored multiple-choice questions, 90 minutes, USD 200 plus applicable taxes. Up to 10 unscored questions may be added, so you can see up to 55; the 90 minutes already allows for them, and you cannot tell which ones they are. All code is Python, and no test aids are allowed. Proctored online or at a test center, in English. Valid for two years.
The official guide names no Spark version, so behavior specific to Spark 4.x is excluded here and defaults that have since changed are flagged. Databricks does not publish a passing score.
Weights below are from the Databricks certification page and match the exam guide updated October 8, 2026, both checked October 27, 2026. Each topic sits in the domain the guide lists it under, which puts a few where you might not look: caching and garbage collection under architecture, broadcast joins and accumulators under the DataFrame API, deploy modes under Spark Connect. For the other Databricks exams and which one to take first, see the list of Databricks certifications.
- Apache Spark architecture and components20%
- Using Spark SQL20%
- Developing Apache Spark DataFrame/DataSet API applications30%
- Troubleshooting and tuning DataFrame API applications10%
- Structured Streaming10%
- Using Spark Connect to deploy applications5%
- Using pandas API on Apache Spark5%
The DataFrame API domain is the largest, and the one where answers hinge on exact method signatures. With limited study time, start there.
Apache Spark architecture and components (20%)
What happens between calling an action and getting a result: who plans the work, who runs it, where memory and cached data live, and how Spark recovers when a machine dies.
| Concept | What to remember |
|---|---|
| Advantages and challenges | Advantages: in-memory, distributed processing, one engine for batch, SQL, streaming, and machine learning, APIs in Python, SQL, Scala, and Java, plans optimized before anything runs, and recovery from lineage. Challenges: shuffles cost network and disk, skew leaves a few slow tasks, memory needs tuning (out-of-memory errors, GC pauses), Python UDFs pay for serialization, and anything collected lands on one driver. |
| Execution hierarchy | Application > Job > Stage > Task. Usually one job per action, though a few actions launch more than one. Stages are bounded by shuffles (wide dependencies). One task per partition, holding one slot for its duration. |
| Executor | A JVM process on a worker that holds slots and runs tasks. Slots are the unit of parallelism, one per core by default (spark.task.cpus is 1). Executors also hold cached data. |
| Driver | Builds the DAG, splits it into stages and tasks, schedules them, collects results. It does not process the data, which is why collect() on a large DataFrame can exhaust driver memory. |
| Cluster manager | Allocates executors to the application (standalone, YARN, Kubernetes, or the Databricks-managed equivalent). It hands out resources; it does not schedule tasks. |
| Memory model | A unified region shared between execution (shuffles, joins, sorts) and storage (cache), sized by spark.memory.fraction: 0.6 of the heap after 300 MiB is reserved. Execution can evict storage, but storage cannot evict execution, so heavy caching gets reclaimed under pressure. Eviction stops at spark.memory.storageFraction (0.5 of the region), below which cached blocks are safe. |
| Garbage collection | Executors are JVMs, so a long GC pause stalls every task on that executor. GC time per task is in the stage detail page, and per executor in the Executors tab, which turns task time red once GC passes 10% of it. The tuning guide's fixes: lower spark.memory.fraction, since caching fewer objects beats slowing tasks down, or try G1GC with -XX:+UseG1GC in spark.executor.extraJavaOptions. GC logging (-verbose:gc) prints to the executor logs, not the driver's. |
| Lazy evaluation | Transformations only build the plan. Nothing runs until an action (count, collect, show, write, take), so a mistake in a transformation surfaces late. |
| Narrow vs wide | Narrow (select, filter, withColumn, union) needs no data movement: each output partition reads one input partition. Wide (groupBy, join, distinct, repartition) shuffles. A withColumn holding a window expression is the exception, and it shuffles. |
| Fault tolerance | Lost partitions are recomputed from lineage, not restored from a replica. DataFrames are immutable, which makes recomputation deterministic. |
| DataFrame vs Dataset | A Dataset is typed, checked at compile time, and exists only in Scala and Java. A DataFrame is a Dataset of rows (Dataset[Row] in Scala). Python has only DataFrames. Both are lazy, immutable, and optimized by Catalyst. |
| SparkSession lifecycle | SparkSession.builder.appName("x").getOrCreate() returns the active session if one exists and applies the builder's SQL settings to it. spark.conf.set on a static setting such as spark.executor.memory raises. spark.newSession() shares the SparkContext and the table cache but has its own temp views, UDFs, and SQL config. spark.stop() stops the underlying context. One SparkContext per JVM. |
cache() and persist() | For DataFrames, cache() equals argument-free persist(), and that default is MEMORY_AND_DISK: partitions that do not fit go to disk. For RDDs the default is MEMORY_ONLY, and partitions that do not fit are recomputed instead. That mismatch is the classic confusion. PySpark spells the DataFrame default MEMORY_AND_DISK_DESER (deserialized), same idea. |
| Storage levels | MEMORY_ONLY recomputes partitions that do not fit rather than spilling. DISK_ONLY skips memory. The _2 suffix replicates each partition on two cluster nodes. |
| Caching lifecycle | Caching is lazy: nothing is stored until an action materializes it, so the first action after cache() pays full cost. unpersist() takes effect right away; its blocking=False default only means the call does not wait for removal. |
| Spark modules | Spark Core (scheduling, memory, fault tolerance, RDDs), Spark SQL and DataFrames, pandas API on Spark, Structured Streaming, and MLlib, which the guide spells "MLib". MLlib's primary API is DataFrame based, in pyspark.ml: an Estimator's fit() returns a Model, and a Model, like any Transformer, adds columns with transform(). The RDD-based pyspark.mllib has been in maintenance mode since Spark 2.0. |
Using Spark SQL (20%)
How a name becomes queryable, how SQL and the DataFrame API meet, how files and JDBC sources are read and written, and a few constructs Spark handles its own way. Managed versus external is the highest-value row here.
| Concept | What to remember |
|---|---|
createOrReplaceTempView("t") | Session scoped. Query as FROM t. Disappears when the session ends, invisible to other sessions. |
createGlobalTempView("t") | Application scoped, shared across sessions in the same application, dropped when the application ends. Must be queried as FROM global_temp.t; the prefix is not optional (name set by spark.sql.globalTempDatabase). |
| What a view is | A named plan, not stored data. Creating one writes nothing to disk and costs nothing until queried. |
spark.sql("...") | Returns a lazy DataFrame, exactly like the API path. Both go through the same Catalyst optimizer, so neither is faster by nature. |
| Parameterized queries | Spark 3.4 and later: spark.sql("... WHERE id = :id", args={"id": 5}). Named markers take a leading colon. Preferred over string concatenation. |
| Reading a table | spark.table("t") or spark.read.table("t"). Both work; spark.read.load() is for paths, not catalog names. |
| SQL on files | SELECT * FROM parquet.`/path`, and likewise json, orc, csv, text, and delta (Delta Lake must be installed, as it is on Databricks). The path takes backticks; single or double quotes are a parse error. text returns one column, value. |
| CSV queried directly | csv.`/path` uses default options: no header, so the header line comes back as a data row, columns are named _c0, _c1, and every column is a string. For options, create a view: CREATE TEMPORARY VIEW v USING csv OPTIONS (path '/path', header 'true', inferSchema 'true'). |
| JDBC reads | spark.read.format("jdbc").option("url", url).option("dbtable", "t"), or option("query", sql), never both. A plain read is one partition over one connection. A parallel read needs all four of partitionColumn, lowerBound, upperBound, and numPartitions; the bounds only set the stride and filter no rows. query cannot be combined with partitionColumn, so put the query in dbtable as an aliased subquery. |
| JDBC writes | df.write.format("jdbc") with the same options. The default mode errors if the table exists. overwrite drops and recreates the table, unless truncate is true, which keeps its definition and indexes. |
UNION vs UNION ALL | SQL UNION deduplicates (and therefore shuffles). UNION ALL keeps duplicates and matches the DataFrame union method. |
| Top-N per group | ROW_NUMBER() OVER (PARTITION BY dept ORDER BY salary DESC) inside a subquery or CTE, then filter to = 1 outside it. A window function cannot go in WHERE. QUALIFY does it in one statement on Databricks SQL, and in Apache Spark only from 4.2, so treat it as out of scope for the exam. |
RANK vs DENSE_RANK | RANK leaves gaps after ties (1, 2, 2, 4). DENSE_RANK does not (1, 2, 2, 3). ROW_NUMBER never ties. |
| CTEs | WITH name AS (query), then the main statement. Multiple CTEs are separated by commas, with WITH written once. |
saveAsTable("t") | Registers a managed table: Spark owns the files, and DROP TABLE deletes the data. Under Unity Catalog the files go after a retention window, within which UNDROP TABLE recovers the table. |
| External tables | Adding .option("path", "/some/path") to saveAsTable makes it external: DROP TABLE removes the metadata and keeps the files. Storage cleanup is on you. |
save("/path") | Writes files and registers nothing in the catalog. No table to query by name. |
| Sorted, partitioned tables | df.write.partitionBy("c").bucketBy(4, "id").sortBy("id").saveAsTable("t"). bucketBy and sortBy work only with saveAsTable: save() or parquet(path) raises, and so does sortBy without bucketBy. Delta tables do not support bucketing. sortWithinPartitions sorts the rows inside each file with no shuffle; orderBy before a write adds one. |
Overwrite with partitionBy | Writes one folder per value (country=US/). By default (spark.sql.sources.partitionOverwriteMode is static), mode("overwrite") deletes every existing partition, including ones absent from the new data. Set it to dynamic to replace only the partitions being written. |
| Save modes in SQL | INSERT INTO t appends and INSERT OVERWRITE t replaces. CREATE TABLE t AS SELECT on an existing name errors, like the writer's default mode; IF NOT EXISTS turns it into a silent no-op, like mode ignore. CREATE OR REPLACE TABLE ... AS SELECT works on Delta and fails on a Parquet table. |
| Catalog API | spark.catalog.listTables(), listDatabases(), tableExists("t"), dropTempView("v"), currentDatabase(). SQL twins: SHOW TABLES, DESCRIBE TABLE. |
explain() | Prints the physical plan only. explain(True) adds the parsed, analyzed, and optimized logical plans. Scan for Exchange (a shuffle) and BroadcastHashJoin. |
Developing Apache Spark DataFrame/DataSet API applications (30%)
The largest domain, and the one where correctness hinges on one detail at a time: name or position, at least or at most, column or string, seconds or milliseconds. Python has no typed Dataset API, so in practice this is the DataFrame API.
| Concept | What to remember |
|---|---|
withColumn | Adds a column, or replaces it if the name exists. The second argument is a Column expression, not a bare Python value, so wrap constants in lit(). |
withColumnRenamed | Renames an existing column and is a no-op if the name is not found. It does not create anything, and it does not raise. |
drop | drop("a", "b") takes several names. A name that does not exist is ignored silently, as with withColumnRenamed. |
selectExpr and expr | Both accept SQL expression strings: selectExpr("value * 2 AS doubled"), expr("value * 2"). Plain select("value * 2") treats the string as a column name and fails to resolve. |
split | split(col("s"), ",") returns an array; take parts with getItem(0) or [0]. The pattern is a Java regex, so splitting on "." returns only empty strings; escape it as "\\.". An index past the end returns null. Version note: with ANSI mode on, the default from Spark 4.0, it raises INVALID_ARRAY_INDEX instead. |
explode | One output row per array element. Rows whose array is null or empty disappear; explode_outer keeps them with a null. posexplode adds a pos column, and exploding a map yields key and value columns. |
sample() | Full signature sample(withReplacement, fraction, seed); PySpark also accepts sample(fraction) and sample(fraction, seed). The fraction is a per-row probability, so the result size is approximate, not an exact row count. |
dropDuplicates() | Takes an optional subset: dropDuplicates(["id"]). Which surviving row you get is not defined. |
distinct() | Whole-row only, no arguments. It is dropDuplicates() with no subset. |
dropna(thresh=n) | Keeps rows with at least n non-null values (the opposite of the common guess). thresh overrides how. Otherwise how="any" (the default) drops a row with any null, how="all" drops only all-null rows, and subset limits which columns count. df.na.drop() is the same method. |
| Null checks | col("x") == None matches nothing: a comparison with null is null, never true. Use isNull() and isNotNull(), or eqNullSafe (SQL <=>) to treat two nulls as equal. |
fillna | Also na.fill. It fills only columns whose type matches the value: fillna(0) fills numeric columns and skips strings, fillna("n/a") the reverse. A dict, fillna({"s": "n/a", "i": -1}), sets a value per column. |
| Counting | df.count() is an action and returns a Python int. Inside agg, count("x") skips nulls, count("*") counts every row, and countDistinct("x") is exact and ignores nulls. mean is an alias of avg, and both skip nulls. |
approx_count_distinct | approx_count_distinct("x", rsd=0.05) estimates with HyperLogLog++; rsd is the maximum relative standard deviation, 5% by default. It is cheaper because each partition ships a small sketch rather than every distinct value: its plan has one Exchange where countDistinct has two. The guide's sample answer says it avoids the costly shuffle; read that as a far smaller one. On a stream, countDistinct raises and points you here. |
describe() and summary() | describe() gives count, mean, stddev, min, and max. summary() adds approximate 25%, 50%, and 75% percentiles and takes a subset, summary("min", "max"). Both return a DataFrame of strings, so call show() to see it. |
| Unix epoch to date string | from_unixtime(col) takes seconds since the epoch and returns a string, yyyy-MM-dd HH:mm:ss in the session time zone, or your pattern as the second argument. Milliseconds must be divided by 1000 first, or you get a year like 55840. timestamp_seconds(col) returns a timestamp instead, and unix_timestamp() goes the other way, to seconds as a long. |
| Parsing and formatting dates | to_date(col, "dd/MM/yyyy") parses a string into a date and returns null when the text does not match. date_format(col, "yyyy-MM") turns a date or timestamp into a string. Pattern letters are case sensitive: MM is month, mm is minutes, and the week-based YYYY raises on Spark 3.5 rather than printing a wrong year. Version note: with ANSI mode on, the default from Spark 4.0, a mismatch raises instead of returning null. |
| Extracting date parts | year, month, dayofmonth, dayofweek, dayofyear, weekofyear, quarter, hour. dayofweek runs 1 = Sunday to 7 = Saturday. They accept a date-like string and cast it. SQL adds extract(YEAR FROM d) and date_part. |
union | Matches columns by position, not name, and keeps duplicates. Equivalent to SQL UNION ALL. Mismatched order produces wrong data with no error whenever the types line up. |
unionByName | Matches by name, and takes allowMissingColumns=True to fill absent columns with null instead of failing. |
| Join keys | a.join(b, ["id", "dt"]) joins on names: an equi-join on both keys that keeps one copy of each. A condition, (a.id == b.id) & (a.dt == b.dt), keeps both copies, and a later select("id") fails as ambiguous unless you name the side, a.id. |
| Join types | how defaults to inner. left, left_outer, and leftouter are one type, filling unmatched right columns with nulls; likewise right, full/outer, semi, anti, cross. An invented name such as left_join raises. a.crossJoin(b) is the Cartesian product: rows of a times rows of b. |
| Left semi join | Left rows with a match, left columns only, no row duplication even if the right side matches many times. It behaves like a filter. |
| Left anti join | Left rows without a match, left columns only, again with no duplication. The natural way to express "not in". |
| Broadcast join | from pyspark.sql.functions import broadcast, then big.join(broadcast(small), "key"). The small side is copied to every executor, so the big side is not shuffled; the plan shows BroadcastHashJoin. broadcast is a function and a hint, not a join type, so it never goes in the how argument. Also small.hint("broadcast"), and in SQL /*+ BROADCAST(t) */ (or BROADCASTJOIN, MAPJOIN). In a left join only the right side can be broadcast; hinting the left side falls back to a sort-merge join. |
| Reader shape | spark.read.format("csv").option("header", True).schema(s).load(path). spark.read is a property, so spark.read() is wrong. Shortcuts like spark.read.csv(path) also exist. |
| Writer shape | df.write.mode("overwrite").partitionBy("col").parquet(path). df.write is also a property. partitionBy takes column names, not a count. |
| Write modes | append, overwrite, ignore, error (the default, also spelled errorifexists). There is no update and no merge mode. |
| Explicit schemas | StructType([StructField("a", IntegerType(), True)]), or a DDL string, "a INT, b STRING". The third argument is nullability. An explicit schema also skips the inference scan, so reads start faster and types stop drifting. |
| Sorting | orderBy and sort are the same method. Descending: desc("x"), col("x").desc(), or ascending=[False, True], one flag per column. There is no descending argument, and PySpark ignores it silently and sorts ascending. Nulls come first ascending, last descending; asc_nulls_last() and its siblings override that. |
| Schema inspection | printSchema() prints a tree and returns None. schema is a property holding a StructType, so schema() raises. dtypes gives (name, type) pairs, and columns a list of names. |
| To and from Python lists | collect() returns a list of Row, read with row.asDict(), row["name"], or row.name. take(n) and head(n) return lists; first() and head() return one Row, or None when empty. There is no toList(). Back again: spark.createDataFrame([(1, "a")], ["id", "name"]); a list of bare values needs a type, spark.createDataFrame([1, 2], "int"), and lands in a column named value. |
| Iterating | toLocalIterator() brings rows to the driver a partition at a time, so it needs memory for the largest partition, not the whole result. foreach(f) runs f on the executors: on a cluster, a print inside it lands in executor logs, not your notebook. |
| Python UDF | Wrap the function with udf(f, ReturnType()) and call the wrapped object in DataFrame code, never the raw Python function. The decorator @udf("string") does the same. To call it from SQL, register it: spark.udf.register("shout", f, "string"). |
| UDF cost | Built-in functions beat Python UDFs: a Python UDF serializes rows to a Python worker process and back, and Catalyst cannot see inside it to optimize or push down. |
| UDFs with and without state | A plain udf or pandas_udf in select or withColumn is stateless: each row on its own, no state store, so a stream that uses it scales out freely. Logic that must remember earlier records, such as running totals or sessions, needs a stateful operator; the custom one in PySpark is groupBy(...).applyInPandasWithState(...), Spark 3.4 and later. mapGroupsWithState and flatMapGroupsWithState are Scala and Java only. |
| State stores | Stateful operators (streaming aggregations, dropDuplicates, stream-stream joins, applyInPandasWithState) keep a versioned key-value store on the executors, saved to the state/ folder of the checkpoint so a restart restores it. The default, HDFSBackedStateStoreProvider, holds state in executor JVM memory, so millions of keys mean GC pauses. RocksDBStateStoreProvider, Spark 3.2 and later, keeps it in native memory and local disk; set it with spark.sql.streaming.stateStore.providerClass. |
| Broadcast variables | Immutable, read-only data shipped once per executor and reused by all its tasks: bc = spark.sparkContext.broadcast(lookup), read as bc.value inside tasks. For lookup tables, not results. Both kinds of shared variable need a SparkContext, so Spark Connect has neither. |
| Accumulators | Driver-readable counters that workers only add to: acc = spark.sparkContext.accumulator(0), acc.add(1) in tasks, acc.value on the driver; reading value inside a task raises. Each task's update applies once when the accumulator is updated inside an action. Inside a transformation, a retried or speculative task can apply it more than once, so counts run high. |
Troubleshooting and tuning DataFrame API applications (10%)
Small on the exam, large in daily work. Learn the defaults by heart, then learn what Adaptive Query Execution now does for you automatically, and where the logs are when it does not.
| Concept | What to remember |
|---|---|
spark.sql.shuffle.partitions | Default 200. Governs the post-shuffle partition count, unrelated to input file count. With AQE on it is a starting point that gets coalesced down at runtime. |
spark.sql.autoBroadcastJoinThreshold | Default 10 MB, expressed in bytes (10485760), applied to the estimated size of the smaller side. Set -1 to disable automatic broadcasting. |
| Setting config | spark.conf.set("spark.sql.shuffle.partitions", 400). The key is a quoted string; the value may be a string or a number. |
| Adaptive Query Execution | On by default since Spark 3.2, driven by runtime statistics collected at shuffle boundaries. Three behaviors to know: coalescing shuffle partitions, switching sort-merge join to broadcast hash join, and splitting skewed partitions. Current docs also list sort-merge to shuffled hash join, so three is not a closed set. |
| Dynamic partition pruning | A separate feature from AQE, added in Spark 3.0. It prunes fact-table partitions using a filter applied on the dimension side of a join. Do not attribute it to AQE. |
repartition(n[, cols]) | Wide, full shuffle, can increase or decrease the count, produces roughly even partitions. Expensive but predictable. Passing columns co-locates matching rows. |
coalesce(n) | Narrow, no full shuffle, decrease only: asking for more partitions than you have leaves the count unchanged. It merges existing partitions, so sizes can end up uneven, and coalesce(1) also throttles upstream stages to that parallelism. Both methods are lazy. |
| Skew symptoms | A few straggler tasks far slower than the median, spills to disk, and uneven shuffle read sizes on the stage detail page. Stage runtime equals the slowest task. |
| Skew fixes | AQE skew join handling, broadcasting the small side, or salting the hot key (random suffix, join, aggregate again). Null keys are a classic skew source, and Spark does not drop them for you. |
| Driver OOM | collect() and toPandas() pull the whole result into one JVM. Use take(n) or show(), or limit() before collecting. spark.driver.maxResultSize (default 1g) aborts an action whose serialized results exceed it, often before the heap runs out; it is fixed at startup, and setting it at runtime raises. |
| Executor OOM | More, smaller partitions, a lower broadcast threshold, or fewer cores per executor so fewer tasks share the heap. |
| Spark UI tabs | Jobs, Stages, Storage, Environment, Executors, SQL, plus Structured Streaming once such a query runs. GC time per task is in the stage detail, spark.executor.memory in Environment, cached DataFrames in Storage. |
| Driver and executor logs | The driver log is wherever the driver runs: your console in client mode, a worker in cluster mode. Executor logs are in the Spark UI: the Executors tab links stdout and stderr for each executor, and the task table on a stage's detail page links to the logs of the executor that ran each task, which is the guide's own sample answer. Driver code logs to the driver; code inside a UDF runs on an executor and logs there. |
| Customizing logs | spark.sparkContext.setLogLevel("WARN") changes the level at runtime; valid levels are ALL, TRACE, DEBUG, INFO, WARN, ERROR, FATAL, and OFF, and anything else raises. The config file is conf/log4j2.properties. Version note: Spark 3.3 moved from Log4j 1.x to 2.x, so older material shows log4j.properties. To keep the UI after the application ends, set spark.eventLog.enabled to true and open the event logs in the History Server. |
| Reading the logs | A driver out-of-memory error, in the driver log, usually follows collect(), toPandas(), or a large broadcast. An executor one shows as java.lang.OutOfMemoryError in that executor's log, and as lost executors and failed tasks in the UI. Underutilization shows as fewer tasks than cores in a stage, or idle executors in the Executors tab; more partitions is the usual fix. |
Structured Streaming (10%)
Which output mode is legal for which query, what the checkpoint stores, where the watermark goes, and how deduplication keeps its state in check. Treat the query as an ever-growing result table and most of the mode rules follow.
| Concept | What to remember |
|---|---|
spark.readStream | Mirrors spark.read in shape. File sources require an explicit schema by default, so a consistent schema survives failures. spark.sql.streaming.schemaInference=true re-enables inference, but treat the explicit schema as the rule. |
| Kafka source | format("kafka") with kafka.bootstrap.servers and subscribe. key and value arrive as binary and need cast("string") before use. |
| Output modes | append (the default: only new result rows, and aggregations need a watermark), complete (the entire result table every trigger, requires an aggregation, keeps all state), update (only rows changed this batch; with no aggregation it behaves like append). overwrite is a batch write mode, absent from streaming. |
| Output sinks | File (parquet, json, csv, orc): append only, exactly-once. Kafka and foreach: append, update, or complete, at-least-once. foreachBatch: as fault tolerant as your function. Console and memory are for debugging and collect output to driver memory; the memory sink is queried as a table named by queryName. toTable("t") writes to a catalog table. |
| Triggers | Default: micro-batch as soon as the previous one finishes. processingTime="10 seconds" for a fixed cadence. availableNow=True drains everything currently available across one or more batches, then stops; it replaces once, deprecated since Spark 3.4. continuous is experimental, map-like operations only, at-least-once. |
| Checkpointing | option("checkpointLocation", path) stores offset ranges and state, not the data. A restart resumes where it stopped. One checkpoint location per query, never shared. |
| Exactly-once | Needs all three: a replayable source (Kafka, files), an idempotent sink (files, Delta, or dedup by batchId in foreachBatch), and checkpointing. The socket source is not fault tolerant; a Kafka sink alone gives at-least-once. |
| Watermarks | withWatermark("eventTime", "10 minutes") must name the same column the aggregation windows on, and must be called before the groupBy, not after. It bounds state, finalizes windows, and drops records later than the threshold. Append-mode aggregations require it; complete mode keeps all state regardless. |
| Window types | window(col, "10 minutes") tumbling. window(col, "10 minutes", "5 minutes") sliding, so one event can land in more than one window. session_window(col, "5 minutes") is gap based. |
| Stateless vs stateful | Stateless: select, filter, withColumn. Stateful (needs a state store, and a watermark to bound it): aggregations, dropDuplicates, stream-stream joins. |
| Streaming deduplication | Without a watermark, dropDuplicates(["guid"]) keeps every key it has ever seen, so state grows forever. With one, put the event-time column in the subset, withWatermark("ts", "1 hour").dropDuplicates(["guid", "ts"]), and state older than the watermark is dropped. When duplicates can carry different event times, Spark 3.5's dropDuplicatesWithinWatermark(["guid"]) deduplicates on the key alone within the delay; it requires a watermark and raises without one. |
| Unsupported operations | The programming guide lists limit and distinct as unsupported and allows orderBy/sort only after an aggregation in complete mode; for the exam, go with that list. The Spark 3.5 engine rejects less: limit fails only in update mode, DataFrame distinct() runs as a streaming dropDuplicates(), and countDistinct raises. Batch actions such as count(), show(), and take() raise on a stream, which runs only through writeStream.start(). dropDuplicates is allowed. Chained stateful operators are unsupported in update and complete mode. |
foreachBatch(f) | Hands f(batchDf, batchId) a normal batch DataFrame: the escape hatch for JDBC writes, several sinks, or a Delta MERGE. At-least-once by default, which is why batchId matters. foreach is per-row. |
| Query handle | query = ....start() returns immediately. awaitTermination() blocks, stop() halts, lastProgress and status report, spark.streams.active lists running queries. |
Using Spark Connect to deploy applications (5%)
The client builds a plan locally and ships it to a server that does the real work. Remember that the client has no SparkContext and the rest follows. The guide also files deploy modes under this domain.
| Concept | What to remember |
|---|---|
| Protocol chain | The client translates DataFrame operations into an unresolved logical plan, encodes it with protocol buffers, and sends it over gRPC. The server holds the real driver, resolves and optimizes the plan, and executes it. Results stream back as Apache Arrow encoded row batches. |
| Connecting | SparkSession.builder.remote("sc://host:15002").getOrCreate(). Default port 15002. Also settable through the SPARK_REMOTE environment variable, the spark.remote config, or --remote. |
| Starting the server | ./sbin/start-connect-server.sh. One server serves many clients, each with an isolated session, so temp views and session configs do not leak between them. |
| Supported | The DataFrame API, Spark SQL, reads and writes, UDFs, and Structured Streaming (since Spark 3.5). |
| Not supported | SparkContext and the RDD API. The protocol has no RDD execution path, and clients cannot reach cluster-wide properties. Anything built on RDDs is off the table. |
| Stated benefits | The docs list three: stability (a client cannot bring down the driver JVM, and dependencies stay separate), upgradability (the server upgrades independently of clients), and debuggability and observability (IDE debugging against a remote cluster). Thin, JVM-free clients are the enabling design, not a fourth named benefit. |
| Deploy modes | The guide names three: client, cluster, and local. cluster: driver on the cluster. client, the spark-submit default: driver on the submitting machine. In both, executors run on worker nodes; only the driver moves. local: driver and executor share one JVM on one machine, threads as slots, so it is the mode where everything runs on a single node; set it with --master local[4] or local[*] for every core. Outside the exam, spark-submit --deploy-mode accepts only client or cluster and rejects cluster with a local master; on the exam, local counts as a deploy mode. |
Using pandas API on Apache Spark (5%)
Conversion boundaries (what collects to the driver, what stays distributed), index behavior with no pandas equivalent, and the vectorized UDF family.
| Concept | What to remember |
|---|---|
| Import and entry points | import pyspark.pandas as ps, then ps.read_csv(...), ps.DataFrame(...). Koalas is the predecessor project, folded into PySpark in 3.2, and a plausible wrong answer. |
| Converting | spark_df.pandas_api(index_col=...) converts and stays distributed; psdf.to_spark() converts back. to_pandas() and toPandas() collect to the driver and can exhaust its memory. |
| Laziness | Execution is lazy, because operations build Spark plans underneath. pandas itself is eager: that is the difference that trips people up. |
| Index types | distributed-sequence (the default): a global, consecutive 0..n-1 index computed distributively, costing an extra pass over the data. sequence: also consecutive, but built on a window with no partitioning, so all rows funnel through one partition, which is dangerous at scale. distributed: monotonically_increasing_id, cheap and increasing but not consecutive and not deterministic. Set with ps.set_option('compute.default_index_type', ...). |
| Ops across two frames | Operations spanning two pandas-on-Spark objects raise by default, because they imply a hidden join on the index. Enable with ps.set_option('compute.ops_on_diff_frames', True). Version note: Spark 4.0 flipped this default to True; on any 3.x runtime, expect the error. The guide names no Spark version, so know both. |
| Vectorized UDFs | @pandas_udf("double") runs vectorized over Arrow batches (up to 10,000 rows each by default), far faster than row-wise udf(). The type hints pick the kind: pd.Series to pd.Series for select and withColumn; Iterator[pd.Series] to Iterator[pd.Series] when expensive setup, such as loading a model, should happen once rather than per batch; pd.Series to a scalar for aggregations in groupBy().agg() and over windows. The older PandasUDFType argument still works on 3.x, with a warning. Built-in functions still beat both. |
applyInPandas | groupBy(...).applyInPandas(f, schema): f receives each whole group as one pandas DataFrame, so the group must fit in executor memory. The output schema argument is required. |
mapInPandas | f receives an iterator of pandas DataFrames per partition and may return a different number of rows. For per-partition work with no grouping. Also takes a schema. |
| When to use it | Existing pandas code, or a pandas-fluent team, with data too big for one machine. For small data plain pandas is faster: no scheduling or serialization overhead. |
Closing quick reference: API shape eliminations
When two answer options differ only in syntax, these rules settle it.
| Rule | What to remember |
|---|---|
| Reader and writer entry points | spark.read is a property, never spark.read(). Same for df.write, spark.readStream, and df.writeStream. |
| Properties vs methods | df.schema, df.columns, df.dtypes, and df.na are properties; printSchema(), collect(), and count() are methods. An option that calls schema() or invents toList() is wrong. |
| Column references | col("x") or the string "x"; df["x"] and df.x also work and disambiguate self-joins. A bare identifier is a Python NameError. |
| Boolean conditions | ==, &, |, ~ with parentheses around each condition, because & binds tighter than ==. Applying Python's and, or, or not to Columns raises. |
| Writer modes | append, overwrite, ignore, error (the default, also errorifexists). Anything else, including update and merge, is not a write mode. |
One last pass
Four things carry the most weight per minute of study: DataFrame API method shapes, the managed versus external consequence on DROP, watermark placement, and the two cache defaults (MEMORY_AND_DISK for DataFrames, MEMORY_ONLY for RDDs). State those cold and the rest is reinforcement.
This is an unofficial, community-maintained reference with no affiliation to Databricks. Exam content, weights, pricing, and logistics change; confirm against the current official exam guide before booking, and check behavior against the docs for your Spark version. Last checked October 27, 2026, against the exam guide updated October 8, 2026; the behavior added in that pass was run on Apache Spark 3.5.3.



