Most questions come down to precise method behavior: argument order, defaults, and which operations shuffle. Each table below pairs a concept with the detail that decides the answer. Read the left column, answer from memory, then check the right one.
The exam: 45 questions, 90 minutes, $200. Proctored online or at a test center. Valid for two years.
The official guide names no Spark version, so behavior specific to Spark 4.x is excluded here and defaults that have since changed are flagged. Databricks does not publish a passing score.
Weights below are from the Databricks certification page, checked August 2026.
- Apache Spark architecture and components20%
- Using Spark SQL20%
- Developing Apache Spark DataFrame/DataSet API applications30%
- Troubleshooting and tuning DataFrame API applications10%
- Structured Streaming10%
- Using Spark Connect to deploy applications5%
- Using pandas API on Apache Spark5%
The DataFrame API domain is the largest, and the one where answers hinge on exact method signatures. With limited study time, start there.
Apache Spark architecture and components (20%)
What happens between calling an action and getting a result: who plans the work, who runs it, and how Spark recovers when a machine dies.
| Concept | What to remember |
|---|---|
| Execution hierarchy | Application > Job > Stage > Task. Usually one job per action, though a few actions launch more than one. Stages are bounded by shuffles (wide dependencies). One task per partition, holding one slot for its duration. |
| Executor | A JVM process on a worker that holds slots and runs tasks. Slots are the unit of parallelism, one per core by default (spark.task.cpus is 1). Executors also hold cached data. |
| Driver | Builds the DAG, splits it into stages and tasks, schedules them, collects results. It does not process the data, which is why collect() on a large DataFrame can exhaust driver memory. |
| Cluster manager | Allocates executors to the application (standalone, YARN, Kubernetes, or the Databricks-managed equivalent). It hands out resources; it does not schedule tasks. |
| Deploy modes | Two of them. cluster: driver on the cluster. client: driver on the submitting machine. Executors run on workers in both; only the driver moves. local is a master setting, not a deploy mode. |
| Lazy evaluation | Transformations only build the plan. Nothing runs until an action (count, collect, show, write, take), so a mistake in a transformation surfaces late. |
| Narrow vs wide | Narrow (select, filter, withColumn, union) needs no data movement: each output partition reads one input partition. Wide (groupBy, join, distinct, repartition) shuffles. A withColumn holding a window expression is the exception, and it shuffles. |
| Fault tolerance | Lost partitions are recomputed from lineage, not restored from a replica. DataFrames are immutable, which makes recomputation deterministic. |
| Broadcast variables | Immutable, read-only data shipped once per executor and reused by all its tasks. For lookup tables, not results. |
| Accumulators | Driver-readable counters that workers only add to. Each task's update applies once when the accumulator is updated inside an action. Inside a transformation, a retried or speculative task can apply it more than once, so counts run high. |
Using Spark SQL (20%)
How a name becomes queryable, how SQL and the DataFrame API meet, and a few constructs Spark handles its own way. Managed versus external is the highest-value row here.
| Concept | What to remember |
|---|---|
createOrReplaceTempView("t") | Session scoped. Query as FROM t. Disappears when the session ends, invisible to other sessions. |
createGlobalTempView("t") | Application scoped, shared across sessions in the same application, dropped when the application ends. Must be queried as FROM global_temp.t; the prefix is not optional (name set by spark.sql.globalTempDatabase). |
| What a view is | A named plan, not stored data. Creating one writes nothing to disk and costs nothing until queried. |
spark.sql("...") | Returns a lazy DataFrame, exactly like the API path. Both go through the same Catalyst optimizer, so neither is faster by nature. |
| Parameterized queries | Spark 3.4 and later: spark.sql("... WHERE id = :id", args={"id": 5}). Named markers take a leading colon. Preferred over string concatenation. |
| Reading a table | spark.table("t") or spark.read.table("t"). Both work; spark.read.load() is for paths, not catalog names. |
UNION vs UNION ALL | SQL UNION deduplicates (and therefore shuffles). UNION ALL keeps duplicates and matches the DataFrame union method. |
| Top-N per group | ROW_NUMBER() OVER (PARTITION BY dept ORDER BY salary DESC) inside a subquery or CTE, then filter to = 1 outside it. A window function cannot go in WHERE. QUALIFY does it in one statement on Databricks SQL, and in Apache Spark only from 4.2, so treat it as out of scope for the exam. |
RANK vs DENSE_RANK | RANK leaves gaps after ties (1, 2, 2, 4). DENSE_RANK does not (1, 2, 2, 3). ROW_NUMBER never ties. |
| CTEs | WITH name AS (query), then the main statement. Multiple CTEs are separated by commas, with WITH written once. |
saveAsTable("t") | Registers a managed table: Spark owns the files, and DROP TABLE deletes the data. Under Unity Catalog the files go after a retention window, within which UNDROP TABLE recovers the table. |
| External tables | Adding .option("path", "/some/path") to saveAsTable makes it external: DROP TABLE removes the metadata and keeps the files. Storage cleanup is on you. |
save("/path") | Writes files and registers nothing in the catalog. No table to query by name. |
| Catalog API | spark.catalog.listTables(), listDatabases(), tableExists("t"), dropTempView("v"), currentDatabase(). SQL twins: SHOW TABLES, DESCRIBE TABLE. |
explain() | Prints the physical plan only. explain(True) adds the parsed, analyzed, and optimized logical plans. Scan for Exchange (a shuffle) and BroadcastHashJoin. |
Developing Apache Spark DataFrame/DataSet API applications (30%)
The largest domain, and the one where correctness hinges on one detail at a time: name or position, at least or at most, column or string. Python has no typed Dataset API, so in practice this is the DataFrame API.
| Concept | What to remember |
|---|---|
withColumn | Adds a column, or replaces it if the name exists. The second argument is a Column expression, not a bare Python value, so wrap constants in lit(). |
withColumnRenamed | Renames an existing column and is a no-op if the name is not found. It does not create anything, and it does not raise. |
union | Matches columns by position, not name, and keeps duplicates. Equivalent to SQL UNION ALL. Mismatched order produces wrong data with no error whenever the types line up. |
unionByName | Matches by name, and takes allowMissingColumns=True to fill absent columns with null instead of failing. |
dropDuplicates() | Takes an optional subset: dropDuplicates(["id"]). Which surviving row you get is not defined. |
distinct() | Whole-row only, no arguments. It is dropDuplicates() with no subset. |
dropna(thresh=n) | Keeps rows with at least n non-null values (the opposite of the common guess). thresh overrides how. Otherwise how="any" (the default) drops a row with any null, how="all" drops only all-null rows, and subset limits which columns count. |
sample() | Full signature sample(withReplacement, fraction, seed); PySpark also accepts sample(fraction) and sample(fraction, seed). The fraction is a per-row probability, so the result size is approximate, not an exact row count. |
| Left semi join | Left rows with a match, left columns only, no row duplication even if the right side matches many times. It behaves like a filter. |
| Left anti join | Left rows without a match, left columns only, again with no duplication. The natural way to express "not in". |
| Reader shape | spark.read.format("csv").option("header", True).schema(s).load(path). spark.read is a property, so spark.read() is wrong. Shortcuts like spark.read.csv(path) also exist. |
| Writer shape | df.write.mode("overwrite").partitionBy("col").parquet(path). df.write is also a property. partitionBy takes column names, not a count. |
| Write modes | append, overwrite, ignore, error (the default, also spelled errorifexists). There is no update and no merge mode. |
| Explicit schemas | StructType([StructField("a", IntegerType(), True)]). The third argument is nullability. An explicit schema also skips the inference scan, so reads start faster and types stop drifting. |
| Python UDF | Wrap the function with udf(f, ReturnType()) and call the wrapped object in DataFrame code, never the raw Python function. The decorator @udf("string") does the same. |
| UDF cost | Built-in functions beat Python UDFs: a Python UDF serializes rows to a Python worker process and back, and Catalyst cannot see inside it to optimize or push down. |
selectExpr and expr | Both accept SQL expression strings: selectExpr("value * 2 AS doubled"), expr("value * 2"). Plain select("value * 2") treats the string as a column name and fails to resolve. |
Troubleshooting and tuning DataFrame API applications (10%)
Small on the exam, large in daily work. Learn the defaults by heart, then learn what Adaptive Query Execution now does for you automatically.
| Concept | What to remember |
|---|---|
spark.sql.shuffle.partitions | Default 200. Governs the post-shuffle partition count, unrelated to input file count. With AQE on it is a starting point that gets coalesced down at runtime. |
spark.sql.autoBroadcastJoinThreshold | Default 10 MB, expressed in bytes (10485760), applied to the estimated size of the smaller side. Set -1 to disable automatic broadcasting. |
| Setting config | spark.conf.set("spark.sql.shuffle.partitions", 400). The key is a quoted string; the value may be a string or a number. |
| Adaptive Query Execution | On by default since Spark 3.2, driven by runtime statistics collected at shuffle boundaries. Three behaviors to know: coalescing shuffle partitions, switching sort-merge join to broadcast hash join, and splitting skewed partitions. Current docs also list sort-merge to shuffled hash join, so three is not a closed set. |
| Dynamic partition pruning | A separate feature from AQE, added in Spark 3.0. It prunes fact-table partitions using a filter applied on the dimension side of a join. Do not attribute it to AQE. |
repartition(n[, cols]) | Wide, full shuffle, can increase or decrease the count, produces roughly even partitions. Expensive but predictable. Passing columns co-locates matching rows. |
coalesce(n) | Narrow, no full shuffle, decrease only: asking for more partitions than you have leaves the count unchanged. It merges existing partitions, so sizes can end up uneven, and coalesce(1) also throttles upstream stages to that parallelism. Both methods are lazy. |
cache() and persist() | For DataFrames, cache() equals argument-free persist(), and that default is MEMORY_AND_DISK: partitions that do not fit go to disk. For RDDs the default is MEMORY_ONLY, and partitions that do not fit are recomputed instead. That mismatch is the classic confusion. PySpark spells the DataFrame default MEMORY_AND_DISK_DESER (deserialized), same idea. |
| Storage levels | MEMORY_ONLY recomputes partitions that do not fit rather than spilling. DISK_ONLY skips memory. The _2 suffix replicates each partition on two cluster nodes. |
| Caching lifecycle | Caching is lazy: nothing is stored until an action materializes it, so the first action after cache() pays full cost. unpersist() takes effect right away; its blocking=False default only means the call does not wait for removal. |
| Broadcast join | from pyspark.sql.functions import broadcast, then big.join(broadcast(small), "key"). broadcast is a function and a hint, not a join type, so it never goes in the how argument. SQL form: /*+ BROADCAST(t) */. |
| Skew symptoms | A few straggler tasks far slower than the median, spills to disk, and uneven shuffle read sizes on the stage detail page. Stage runtime equals the slowest task. |
| Skew fixes | AQE skew join handling, broadcasting the small side, or salting the hot key (random suffix, join, aggregate again). Null keys are a classic skew source, and Spark does not drop them for you. |
| Driver OOM | collect() and toPandas() pull the whole result into one JVM. Use take(n) or show(), or limit() before collecting. |
| Executor OOM | More, smaller partitions, a lower broadcast threshold, or fewer cores per executor so fewer tasks share the heap. |
| Spark UI tabs | Jobs, Stages, Storage, Environment, Executors, SQL, plus Structured Streaming once such a query runs. GC time per task is in the stage detail, spark.executor.memory in Environment, cached DataFrames in Storage. |
| Memory model | A unified region shared between execution (shuffles, joins, sorts) and storage (cache). Execution can evict storage, but storage cannot evict execution, so heavy caching gets reclaimed under pressure. |
Structured Streaming (10%)
Which output mode is legal for which query, what the checkpoint stores, and where the watermark goes. Treat the query as an ever-growing result table and most of the mode rules follow.
| Concept | What to remember |
|---|---|
spark.readStream | Mirrors spark.read in shape. File sources require an explicit schema by default, so a consistent schema survives failures. spark.sql.streaming.schemaInference=true re-enables inference, but treat the explicit schema as the rule. |
| Kafka source | format("kafka") with kafka.bootstrap.servers and subscribe. key and value arrive as binary and need cast("string") before use. |
| Output modes | append (the default: only new result rows, and aggregations need a watermark), complete (the entire result table every trigger, requires an aggregation, keeps all state), update (only rows changed this batch; with no aggregation it behaves like append). overwrite is a batch write mode, absent from streaming. |
| Triggers | Default: micro-batch as soon as the previous one finishes. processingTime="10 seconds" for a fixed cadence. availableNow=True drains everything currently available across one or more batches, then stops; it replaces once, deprecated since Spark 3.4. continuous is experimental, map-like operations only, at-least-once. |
| Checkpointing | option("checkpointLocation", path) stores offset ranges and state, not the data. A restart resumes where it stopped. One checkpoint location per query, never shared. |
| Exactly-once | Needs all three: a replayable source (Kafka, files), an idempotent sink (files, Delta, or dedup by batchId in foreachBatch), and checkpointing. The socket source is not fault tolerant; a Kafka sink alone gives at-least-once. |
| Watermarks | withWatermark("eventTime", "10 minutes") must name the same column the aggregation windows on, and must be called before the groupBy, not after. It bounds state, finalizes windows, and drops records later than the threshold. Append-mode aggregations require it; complete mode keeps all state regardless. |
| Window types | window(col, "10 minutes") tumbling. window(col, "10 minutes", "5 minutes") sliding, so one event can land in more than one window. session_window(col, "5 minutes") is gap based. |
| Stateless vs stateful | Stateless: select, filter, withColumn. Stateful (needs a state store, and a watermark to bound it): aggregations, dropDuplicates, stream-stream joins. |
| Unsupported operations | No limit, no distinct, and orderBy/sort only after an aggregation and only in complete mode. dropDuplicates is allowed, and Spark 3.5 adds dropDuplicatesWithinWatermark. Chained stateful operators are unsupported in update and complete mode. |
foreachBatch(f) | Hands f(batchDf, batchId) a normal batch DataFrame: the escape hatch for JDBC writes, several sinks, or a Delta MERGE. At-least-once by default, which is why batchId matters. foreach is per-row. |
| Query handle | query = ....start() returns immediately. awaitTermination() blocks, stop() halts, lastProgress and status report, spark.streams.active lists running queries. |
Using Spark Connect to deploy applications (5%)
The client builds a plan locally and ships it to a server that does the real work. Remember that the client has no SparkContext and the rest follows.
| Concept | What to remember |
|---|---|
| Protocol chain | The client translates DataFrame operations into an unresolved logical plan, encodes it with protocol buffers, and sends it over gRPC. The server holds the real driver, resolves and optimizes the plan, and executes it. Results stream back as Apache Arrow encoded row batches. |
| Connecting | SparkSession.builder.remote("sc://host:15002").getOrCreate(). Default port 15002. Also settable through the SPARK_REMOTE environment variable, the spark.remote config, or --remote. |
| Starting the server | ./sbin/start-connect-server.sh. One server serves many clients, each with an isolated session, so temp views and session configs do not leak between them. |
| Supported | The DataFrame API, Spark SQL, reads and writes, UDFs, and Structured Streaming (since Spark 3.5). |
| Not supported | SparkContext and the RDD API. The protocol has no RDD execution path, and clients cannot reach cluster-wide properties. Anything built on RDDs is off the table. |
| Stated benefits | The docs list three: stability (a client cannot bring down the driver JVM, and dependencies stay separate), upgradability (the server upgrades independently of clients), and debuggability and observability (IDE debugging against a remote cluster). Thin, JVM-free clients are the enabling design, not a fourth named benefit. |
Using pandas API on Apache Spark (5%)
Conversion boundaries (what collects to the driver, what stays distributed), index behavior with no pandas equivalent, and the vectorized UDF family.
| Concept | What to remember |
|---|---|
| Import and entry points | import pyspark.pandas as ps, then ps.read_csv(...), ps.DataFrame(...). Koalas is the predecessor project, folded into PySpark in 3.2, and a plausible wrong answer. |
| Converting | spark_df.pandas_api(index_col=...) converts and stays distributed; psdf.to_spark() converts back. to_pandas() and toPandas() collect to the driver and can exhaust its memory. |
| Laziness | Execution is lazy, because operations build Spark plans underneath. pandas itself is eager: that is the difference that trips people up. |
| Index types | distributed-sequence (the default): a global, consecutive 0..n-1 index computed distributively, costing an extra pass over the data. sequence: also consecutive, but built on a window with no partitioning, so all rows funnel through one partition , dangerous at scale. distributed: monotonically_increasing_id, cheap and increasing but not consecutive and not deterministic. Set with ps.set_option('compute.default_index_type', ...). |
| Ops across two frames | Operations spanning two pandas-on-Spark objects raise by default, because they imply a hidden join on the index. Enable with ps.set_option('compute.ops_on_diff_frames', True). Version note: Spark 4.0 flipped this default to True; on any 3.x runtime, which is what the exam assumes, expect the error. |
| Vectorized UDFs | @pandas_udf("double") on a pd.Series to pd.Series function runs vectorized over Arrow batches, far faster than row-wise udf(). Built-in functions still beat both. |
applyInPandas | groupBy(...).applyInPandas(f, schema): f receives each whole group as one pandas DataFrame, so the group must fit in executor memory. The output schema argument is required. |
mapInPandas | f receives an iterator of pandas DataFrames per partition and may return a different number of rows. For per-partition work with no grouping. Also takes a schema. |
| When to use it | Existing pandas code, or a pandas-fluent team, with data too big for one machine. For small data plain pandas is faster: no scheduling or serialization overhead. |
Closing quick reference: API shape eliminations
When two answer options differ only in syntax, these rules settle it.
| Rule | What to remember |
|---|---|
| Reader and writer entry points | spark.read is a property, never spark.read(). Same for df.write, spark.readStream, and df.writeStream. |
| Column references | col("x") or the string "x"; df["x"] and df.x also work and disambiguate self-joins. A bare identifier is a Python NameError. |
| Boolean conditions | ==, &, |, ~ with parentheses around each condition, because & binds tighter than ==. Applying Python's and, or, or not to Columns raises. |
| Writer modes | append, overwrite, ignore, error (the default, also errorifexists). Anything else, including update and merge, is not a write mode. |
One last pass
Four things carry the most weight per minute of study: DataFrame API method shapes, the managed versus external consequence on DROP, watermark placement, and the two cache defaults (MEMORY_AND_DISK for DataFrames, MEMORY_ONLY for RDDs). State those cold and the rest is reinforcement.
This is an unofficial, community-maintained reference with no affiliation to Databricks. Exam content, weights, pricing, and logistics change; confirm against the current official exam guide before booking, and check behavior against the docs for your Spark version. Last checked August 2026.