This is a common PySpark (Transformations & Actions) interview and practice question. Below is the full answer with a clear, step-by-step explanation.

Question

Problem Given a PySpark DataFrame df with columns name, department, salary , write code to select only employees in the "Engineering" department with salary above 70000, returning just their name and salary.

Answer & Explanation

What you need to do

Filter the rows – keep only those where the department is “Engineering” and the salary is greater than 70 000.
Project the columns you want – after the filter, return just the employee’s name and salary.

The code

result = df.filter((df.department == "Engineering") & (df.salary > 70000)) \
           .select("name", "salary")

Why this works

df.filter(...) (or equivalently df.where(...)) takes a Boolean expression and returns a new DataFrame that contains only the rows for which the expression evaluates to true.
The expression you pass must combine the two conditions:
- df.department == "Engineering" checks the department column.
- df.salary > 70000 checks the salary column.
In PySpark you combine Boolean conditions with the bitwise operators & (AND) and | (OR). These operators have higher precedence than the comparison operators (==, >, etc.). If you omit the parentheses around each condition, Python will evaluate the & before the comparisons, producing a completely different (and usually invalid) expression. Therefore each individual condition must be wrapped in its own parentheses.
After the filter, .select("name", "salary") tells Spark to keep only those two columns in the final result. The order of the column names in select determines the order in the output DataFrame.

How you could arrive at this solution

Identify the required rows – read the problem statement and translate “employees in the Engineering department with salary above 70 000” into two logical predicates.
Choose the right DataFrame operation – filter (or where) is the Spark method for row‑level predicates.
Combine the predicates – remember that Spark uses & for logical AND. Write each predicate in parentheses to avoid precedence bugs.
Pick the output columns – after you have the correct rows, use select to keep only the fields you need (name and salary).

Common pitfalls to avoid

Missing parentheses: df.filter(df.department == "Engineering" & df.salary > 70000) will raise a TypeError because Python tries to evaluate df.department == ("Engineering" & df.salary).
Using Python’s and/or: Those operators work on plain Python booleans, not on Spark column objects. Always use & and | when dealing with Spark columns.
Forgetting to import column functions: If you prefer the functional style, you could write from pyspark.sql.functions import col and then df.filter((col("department") == "Engineering") & (col("salary") > 70000)). The logic is identical; just a different way to reference columns.

Putting it all together

# Option 1 – attribute style (shown above)
result = df.filter((df.department == "Engineering") & (df.salary > 70000)) \
           .select("name", "salary")

# Option 2 – using col() for the same logic
from pyspark.sql.functions import col
result = df.filter((col("department") == "Engineering") & (col("salary") > 70000)) \
           .select("name", "salary")

Both snippets produce a DataFrame that contains only the names and salaries of engineers earning more than 70 000. Use whichever style you find clearer; the underlying Spark execution plan is the same.