Value Members

1.  val caseInsensitiveResolution: (String, String) => Boolean
2.  val caseSensitiveResolution: (String, String) => Boolean
3.  def withPosition[A](t: TreeNode[_])(f: => A): A

   Catches any AnalysisExceptions thrown by f and attaches t's position if any.

4.  object AnalysisContext extends Serializable
5.  object AnsiTypeCoercion extends TypeCoercionBase

   In Spark ANSI mode, the type coercion rules are based on the type precedence lists of the input data types.

   In Spark ANSI mode, the type coercion rules are based on the type precedence lists of the input data types. As per the section "Type precedence list determination" of "ISO/IEC 9075-2:2011 Information technology - Database languages - SQL - Part 2: Foundation (SQL/Foundation)", the type precedence lists of primitive data types are as following: * Byte: Byte, Short, Int, Long, Decimal, Float, Double * Short: Short, Int, Long, Decimal, Float, Double * Int: Int, Long, Decimal, Float, Double * Long: Long, Decimal, Float, Double * Decimal: Float, Double, or any wider Numeric type * Float: Float, Double * Double: Double * String: String * Date: Date, Timestamp * Timestamp: Timestamp * Binary: Binary * Boolean: Boolean * Interval: Interval As for complex data types, Spark will determine the precedent list recursively based on their sub-types and nullability.

   With the definition of type precedent list, the general type coercion rules are as following: * Data type S is allowed to be implicitly cast as type T iff T is in the precedence list of S * Comparison is allowed iff the data type precedence list of both sides has at least one common element. When evaluating the comparison, Spark casts both sides as the tightest common data type of their precedent lists. * There should be at least one common data type among all the children's precedence lists for the following operators. The data type of the operator is the tightest common precedent data type. * In * Except * Intersect * Greatest * Least * Union * If * CaseWhen * CreateArray * Array Concat * Sequence * MapConcat * CreateMap * For complex types (struct, array, map), Spark recursively looks into the element type and applies the rules above. Note: this new type coercion system will allow implicit converting String type as other primitive types, in case of breaking too many existing Spark SQL queries. This is a special rule and it is not from the ANSI SQL standard.

6.  object BindParameters extends Rule[LogicalPlan] with QueryErrorsBase

   Finds all named parameters in ParameterizedQuery and substitutes them by literals from the user-specified arguments.

7.  object CTESubstitution extends Rule[LogicalPlan]

   Analyze WITH nodes and substitute child plan with CTE references or CTE definitions depending on the conditions below: 1.

   Analyze WITH nodes and substitute child plan with CTE references or CTE definitions depending on the conditions below: 1. If in legacy mode, or if the query is a SQL command or DML statement, replace with CTE definitions, i.e., inline CTEs. 2. Otherwise, replace with CTE references CTERelationRefs. The decision to inline or not inline will be made later by the rule InlineCTE after query analysis.

   All the CTE definitions that are not inlined after this substitution will be grouped together under one WithCTE node for each of the main query and the subqueries. Any of the main query or the subqueries that do not contain CTEs or have had all CTEs inlined will obviously not have any WithCTE nodes. If any though, the WithCTE node will be in the same place as where the outermost With node once was.

   The CTE definitions in a WithCTE node are kept in the order of how they have been resolved. That means the CTE definitions are guaranteed to be in topological order base on their dependency for any valid CTE query (i.e., given CTE definitions A and B with B referencing A, A is guaranteed to appear before B). Otherwise, it must be an invalid user query, and an analysis exception will be thrown later by relation resolving rules.

8.  object CleanupAliases extends Rule[LogicalPlan] with AliasHelper

   Cleans up unnecessary Aliases inside the plan.

   Cleans up unnecessary Aliases inside the plan. Basically we only need Alias as a top level expression in Project(project list) or Aggregate(aggregate expressions) or Window(window expressions). Notice that if an expression has other expression parameters which are not in its children, e.g. RuntimeReplaceable, the transformation for Aliases in this rule can't work for those parameters.

9.  object DecimalPrecision extends Rule[LogicalPlan] with TypeCoercionRule

   Calculates and propagates precision for fixed-precision decimals.

   Calculates and propagates precision for fixed-precision decimals. Hive has a number of rules for this based on the SQL standard and MS SQL: https://cwiki.apache.org/confluence/download/attachments/27362075/Hive_Decimal_Precision_Scale_Support.pdf https://msdn.microsoft.com/en-us/library/ms190476.aspx

   In particular, if we have expressions e1 and e2 with precision/scale p1/s2 and p2/s2 respectively, then the following operations have the following precision / scale:

   Operation Result Precision Result Scale ------------------------------------------------------------------------ e1 union e2 max(s1, s2) + max(p1-s1, p2-s2) max(s1, s2)

   To implement the rules for fixed-precision types, we introduce casts to turn them to unlimited precision, do the math on unlimited-precision numbers, then introduce casts back to the required fixed precision. This allows us to do all rounding and overflow handling in the cast-to-fixed-precision operator.

   In addition, when mixing non-decimal types with decimals, we use the following rules: - BYTE gets turned into DECIMAL(3, 0) - SHORT gets turned into DECIMAL(5, 0) - INT gets turned into DECIMAL(10, 0) - LONG gets turned into DECIMAL(20, 0) - FLOAT and DOUBLE cause fixed-length decimals to turn into DOUBLE - Literals INT and LONG get turned into DECIMAL with the precision strictly needed by the value

10.  object DeduplicateRelations extends Rule[LogicalPlan]
11.  object EliminateEventTimeWatermark extends Rule[LogicalPlan]

    Ignore event time watermark in batch query, which is only supported in Structured Streaming.

    Ignore event time watermark in batch query, which is only supported in Structured Streaming. TODO: add this rule into analyzer rule list.

12.  object EliminateSubqueryAliases extends Rule[LogicalPlan]

    Removes SubqueryAlias operators from the plan.

    Removes SubqueryAlias operators from the plan. Subqueries are only required to provide scoping information for attributes and can be removed once analysis is complete.

13.  object EliminateUnions extends Rule[LogicalPlan]

    Removes Union operators from the plan if it just has one child.

14.  object EliminateView extends Rule[LogicalPlan] with CastSupport

    This rule removes View operators from the plan.

    This rule removes View operators from the plan. The operator is respected till the end of analysis stage because we want to see which part of an analyzed logical plan is generated from a view.

15.  object EmptyFunctionRegistry extends EmptyFunctionRegistryBase[Expression] with FunctionRegistry
16.  object EmptyTableFunctionRegistry extends EmptyFunctionRegistryBase[LogicalPlan] with TableFunctionRegistry
17.  object FakeSystemCatalog extends CatalogPlugin
18.  object FakeV2SessionCatalog extends TableCatalog with FunctionCatalog
19.  object FunctionRegistry
20.  object FunctionRegistryBase
21.  object GlobalTempView extends ViewType

    GlobalTempView means cross-session global temporary views.

    GlobalTempView means cross-session global temporary views. Its lifetime is the lifetime of the Spark application, i.e. it will be automatically dropped when the application terminates. It's tied to a system preserved database global_temp, and we must use the qualified name to refer a global temp view, e.g. SELECT * FROM global_temp.view1.

22.  object HintErrorLogger extends HintErrorHandler with Logging

    The hint error handler that logs warnings for each hint error.

23.  object KeepLegacyOutputs extends Rule[LogicalPlan]

    A rule for keeping the SQL command's legacy outputs.

24.  object LocalTempView extends ViewType

    LocalTempView means session-scoped local temporary views.

    LocalTempView means session-scoped local temporary views. Its lifetime is the lifetime of the session that created it, i.e. it will be automatically dropped when the session terminates. It's not tied to any databases, i.e. we can't use db1.view1 to reference a local temporary view.

25.  object PersistedView extends ViewType

    PersistedView means cross-session persisted views.

    PersistedView means cross-session persisted views. Persisted views stay until they are explicitly dropped by user command. It's always tied to a database, default to the current database if not specified.

    Note that, Existing persisted view with the same name are not visible to the current session while the local temporary view exists, unless the view name is qualified by database.

26.  object PullOutNondeterministic extends Rule[LogicalPlan]

    Pulls out nondeterministic expressions from LogicalPlan which is not Project or Filter, put them into an inner Project and finally project them away at the outer Project.

27.  object RemoveTempResolvedColumn extends Rule[LogicalPlan]

    The rule ResolveReferences in the main resolution batch creates TempResolvedColumn in UnresolvedHaving/Filter/Sort to hold the temporarily resolved column with agg.child.

    The rule ResolveReferences in the main resolution batch creates TempResolvedColumn in UnresolvedHaving/Filter/Sort to hold the temporarily resolved column with agg.child.

    If the expression hosting TempResolvedColumn is fully resolved, the rule ResolveAggregationFunctions will - Replace TempResolvedColumn with AttributeReference if it's inside aggregate functions or grouping expressions. - Mark TempResolvedColumn as hasTried if not inside aggregate functions or grouping expressions, hoping other rules can re-resolve it. ResolveReferences will re-resolve TempResolvedColumn if hasTried is true, and keep it unchanged if the resolution fails. We should turn it back to UnresolvedAttribute so that the analyzer can report missing column error later.

    If the expression hosting TempResolvedColumn is not resolved, TempResolvedColumn will remain with hasTried as false. We should strip TempResolvedColumn, so that users can see the reason why the expression is not resolved, e.g. type mismatch.

28.  object ResolveCommandsWithIfExists extends Rule[LogicalPlan]

    A rule for handling commands when the table or temp view is not resolved.

    A rule for handling commands when the table or temp view is not resolved. These commands support a flag, "ifExists", so that they do not fail when a relation is not resolved. If the "ifExists" flag is set to true. the plan is resolved to NoopCommand,

29.  object ResolveExpressionsWithNamePlaceholders extends Rule[LogicalPlan]

    Resolve expressions if they contains NamePlaceholders.

30.  object ResolveHints

    Collection of rules related to hints.

    Collection of rules related to hints. The only hint currently available is join strategy hint.

    Note that this is separately into two rules because in the future we might introduce new hint rules that have different ordering requirements from join strategies.

31.  object ResolveInlineTables extends Rule[LogicalPlan] with CastSupport with AliasHelper

    An analyzer rule that replaces UnresolvedInlineTable with LocalRelation.

32.  object ResolveLambdaVariables extends Rule[LogicalPlan]

    Resolve the lambda variables exposed by a higher order functions.

    Resolve the lambda variables exposed by a higher order functions.

    This rule works in two steps: [1]. Bind the anonymous variables exposed by the higher order function to the lambda function's arguments; this creates named and typed lambda variables. The argument names are checked for duplicates and the number of arguments are checked during this step. [2]. Resolve the used lambda variables used in the lambda function's function expression tree. Note that we allow the use of variables from outside the current lambda, this can either be a lambda function defined in an outer scope, or a attribute in produced by the plan's child. If names are duplicate, the name defined in the most inner scope is used.

33.  object ResolveLateralColumnAliasReference extends Rule[LogicalPlan]

    This rule is the second phase to resolve lateral column alias.

    This rule is the second phase to resolve lateral column alias.

    Resolve lateral column alias, which references the alias defined previously in the SELECT list. Plan-wise, it handles two types of operators: Project and Aggregate. - in Project, pushing down the referenced lateral alias into a newly created Project, resolve the attributes referencing these aliases - in Aggregate, inserting the Project node above and falling back to the resolution of Project.

    The whole process is generally divided into two phases: 1) recognize resolved lateral alias, wrap the attributes referencing them with LateralColumnAliasReference 2) when the whole operator is resolved, or contains Window but have all other resolved, For Project, it unwrap LateralColumnAliasReference, further resolves the attributes and push down the referenced lateral aliases. For Aggregate, it goes through the whole aggregation list, extracts the aggregation expressions and grouping expressions to keep them in this Aggregate node, and add a Project above with the original output. It doesn't do anything on LateralColumnAliasReference, but completely leave it to the Project in the future turns of this rule.

    ** Example for Project: Before rewrite: Project [age AS a, 'a + 1] +- Child

    After phase 1: Project [age AS a, lca(a) + 1] +- Child

    After phase 2: Project [a, a + 1] +- Project [child output, age AS a] +- Child

    ** Example for Aggregate: Before rewrite: Aggregate [dept#14] [dept#14 AS a#12, 'a + 1, avg(salary#16) AS b#13, 'b + avg(bonus#17)] +- Child [dept#14,name#15,salary#16,bonus#17]

    After phase 1: Aggregate [dept#14] [dept#14 AS a#12, lca(a) + 1, avg(salary#16) AS b#13, lca(b) + avg(bonus#17)] +- Child [dept#14,name#15,salary#16,bonus#17]

    After phase 2: Project [dept#14 AS a#12, lca(a) + 1, avg(salary)#26 AS b#13, lca(b) + avg(bonus)#27] +- Aggregate [dept#14] [avg(salary#16) AS avg(salary)#26, avg(bonus#17) AS avg(bonus)#27,dept#14] +- Child [dept#14,name#15,salary#16,bonus#17]

    Now the problem falls back to the lateral alias resolution in Project. After future rounds of this rule: Project [a#12, a#12 + 1, b#13, b#13 + avg(bonus)#27] +- Project [dept#14 AS a#12, avg(salary)#26 AS b#13] +- Aggregate [dept#14] [avg(salary#16) AS avg(salary)#26, avg(bonus#17) AS avg(bonus)#27, dept#14] +- Child [dept#14,name#15,salary#16,bonus#17]

    ** Example for Window: Query: select dept as d, sum(salary) as s, avg(s) over (partition by s order by d) as avg from employee group by dept

    After phase 1: 'Aggregate [dept#17], [dept#17 AS d#15, sum(salary#19) AS s#16L, avg(lca(s#16L)) windowspecdefinition(lca(s#16L), lca(d#15) ASC NULLS FIRST, specifiedwindowframe(..)) AS avg#25] +- Relation spark_catalog.default.employee[dept#17,name#18,salary#19,bonus#20,properties#21] It is similar to a regular Aggregate. All expressions in it are resolved, but itself is unresolved due to the Window expression. The rule allows appliction on this case.

    After phase 2: 'Project [dept#17 AS d#15, sum(salary)#26L AS s#16L, avg(lca(s#16L)) windowspecdefinition(lca(s#16L), lca(d#15) ASC NULLS FIRST, specifiedwindowframe(..)) AS avg#25] +- Aggregate [dept#17], [dept#17, sum(salary#19) AS sum(salary)#26L] +- Relation spark_catalog.default.employee[dept#17,name#18,salary#19,bonus#20,properties#21] Same as Aggregate, it extracts grouping expressions and aggregate functions. Window expressions are completely lifted up to upper Project, free from the current Aggregate.

    Then this rule will apply on the Project, adding another Project below. Till this phase, all lateral column alias references have been resolved and removed. Finally, rule ExtractWindowExpressions will apply on the top Project with window expressions. It is guaranteed that ResolveLateralColumnAliasReference is applied before ExtractWindowExpressions.

34.  object ResolvePartitionSpec extends Rule[LogicalPlan]

    Resolve UnresolvedPartitionSpec to ResolvedPartitionSpec in partition related commands.

35.  object ResolveReferencesInAggregate extends SQLConfHelper with ColumnResolutionHelper with AliasHelper

    A virtual rule to resolve UnresolvedAttribute in Aggregate.

    A virtual rule to resolve UnresolvedAttribute in Aggregate. It's only used by the real rule ResolveReferences. The column resolution order for Aggregate is: 1. Resolves the columns to AttributeReference with the output of the child plan. This includes metadata columns as well. 2. Resolves the columns to a literal function which is allowed to be invoked without braces, e.g. SELECT col, current_date FROM t. 3. If aggregate expressions are all resolved, resolve GROUP BY alias and GROUP BY ALL. 3.1. If the grouping expressions contain an unresolved column whose name matches an alias in the SELECT list, resolves that unresolved column to the alias. This is to support SQL pattern like SELECT a + b AS c, max(col) FROM t GROUP BY c. 3.2. If the grouping expressions only have one single unresolved column named 'ALL', expanded it to include all non-aggregate columns in the SELECT list. This is to support SQL pattern like SELECT col1, col2, agg_expr(...) FROM t GROUP BY ALL. 4. Resolves the columns in aggregate expressions to LateralColumnAliasReference if it references the alias defined previously in the SELECT list. The rule ResolveLateralColumnAliasReference will further resolve LateralColumnAliasReference and rewrite the plan. This is to support SQL pattern like SELECT col1 + 1 AS x, x + 1 AS y, y + 1 AS z FROM t. 5. Resolves the columns to outer references with the outer plan if we are resolving subquery expressions.

36.  object ResolveReferencesInSort extends SQLConfHelper with ColumnResolutionHelper

    A virtual rule to resolve UnresolvedAttribute in Sort.

    A virtual rule to resolve UnresolvedAttribute in Sort. It's only used by the real rule ResolveReferences. The column resolution order for Sort is: 1. Resolves the column to AttributeReference with the output of the child plan. This includes metadata columns as well. 2. Resolves the column to a literal function which is allowed to be invoked without braces, e.g. SELECT col, current_date FROM t. 3. If the child plan is Aggregate, resolves the column to TempResolvedColumn with the output of Aggregate's child plan. This is to allow Sort to host grouping expressions and aggregate functions, which can be pushed down to the Aggregate later. For example, SELECT max(a) FROM t GROUP BY b ORDER BY min(a). 4. Resolves the column to AttributeReference with the output of a descendant plan node. Spark will propagate the missing attributes from the descendant plan node to the Sort node. This is to allow users to ORDER BY columns that are not in the SELECT clause, which is widely supported in other SQL dialects. For example, SELECT a FROM t ORDER BY b. 5. If the order by expressions only have one single unresolved column named ALL, expanded it to include all columns in the SELECT list. This is to support SQL pattern like SELECT col1, col2 FROM t ORDER BY ALL. This should also support specifying asc/desc, and nulls first/last. 6. Resolves the column to outer references with the outer plan if we are resolving subquery expressions.

    Note, 3 and 4 are actually orthogonal. If the child plan is Aggregate, 4 can only resolve columns as the grouping columns, which is completely covered by 3.

37.  object ResolveTimeZone extends Rule[LogicalPlan]

    Replace TimeZoneAwareExpression without timezone id by its copy with session local time zone.

38.  object ResolveUnion extends Rule[LogicalPlan]

    Resolves different children of Union to a common set of columns.

39.  object ResolveWindowTime extends Rule[LogicalPlan]

    Resolves the window_time expression which extracts the correct window time from the window column generated as the output of the window aggregating operators.

    Resolves the window_time expression which extracts the correct window time from the window column generated as the output of the window aggregating operators. The window column is of type struct { start: TimestampType, end: TimestampType }. The correct representative event time of a window is window.end - 1.

40.  object ResolveWithCTE extends Rule[LogicalPlan]

    Updates CTE references with the resolve output attributes of corresponding CTE definitions.

41.  object ResolvedTable extends Serializable
42.  object RewriteDeleteFromTable extends Rule[LogicalPlan] with RewriteRowLevelCommand

    A rule that rewrites DELETE operations using plans that operate on individual or groups of rows.

    A rule that rewrites DELETE operations using plans that operate on individual or groups of rows.

    If a table implements SupportsDeleteV2 and SupportsRowLevelOperations, this rule will still rewrite the DELETE operation but the optimizer will check whether this particular DELETE statement can be handled by simply passing delete filters to the connector. If so, the optimizer will discard the rewritten plan and will allow the data source to delete using filters.

43.  object SessionWindowing extends Rule[LogicalPlan]

    Maps a time column to a session window.

44.  object SimpleAnalyzer extends Analyzer

    A trivial Analyzer with a dummy SessionCatalog and EmptyTableFunctionRegistry.

    A trivial Analyzer with a dummy SessionCatalog and EmptyTableFunctionRegistry. Used for testing when all relations are already filled in and the analyzer needs only to resolve attribute references.

    Built-in function registry is set for Spark Connect project to test unresolved functions.

45.  object StreamingJoinHelper extends PredicateHelper with Logging

    Helper object for stream joins.

    Helper object for stream joins. See StreamingSymmetricHashJoinExec in SQL for more details.

46.  object SubstituteUnresolvedOrdinals extends Rule[LogicalPlan]

    Replaces ordinal in 'order by' or 'group by' with UnresolvedOrdinal expression.

47.  object TableFunctionRegistry
48.  object TableOutputResolver
49.  object TimeTravelSpec
50.  object TimeWindowing extends Rule[LogicalPlan]

    Maps a time column to multiple time windows using the Expand operator.

    Maps a time column to multiple time windows using the Expand operator. Since it's non-trivial to figure out how many windows a time column can map to, we over-estimate the number of windows and filter out the rows where the time column is not inside the time window.

51.  object TypeCheckResult
52.  object TypeCoercion extends TypeCoercionBase

    A collection of Rule that can be used to coerce differing types that participate in operations into compatible ones.

    A collection of Rule that can be used to coerce differing types that participate in operations into compatible ones.

    Notes about type widening / tightest common types: Broadly, there are two cases when we need to widen data types (e.g. union, binary comparison). In case 1, we are looking for a common data type for two or more data types, and in this case no loss of precision is allowed. Examples include type inference in JSON (e.g. what's the column's data type if one row is an integer while the other row is a long?). In case 2, we are looking for a widened data type with some acceptable loss of precision (e.g. there is no common type for double and decimal because double's range is larger than decimal, and yet decimal is more precise than double, but in union we would cast the decimal into double).

53.  object UnresolvedAttribute extends Serializable
54.  object UnresolvedFunction extends Serializable
55.  object UnresolvedRelation extends Serializable
56.  case object UnresolvedSeed extends LeafExpression with Unevaluable with Product with Serializable

    A place holder expression used in random functions, will be replaced after analyze.

57.  object UnresolvedTVFAliases extends Serializable
58.  object UnresolvedTableValuedFunction extends Serializable
59.  object UnsupportedOperationChecker extends Logging

    Analyzes the presence of unsupported operations in a logical plan.

60.  object UpdateAttributeNullability extends Rule[LogicalPlan]

    Updates nullability of Attributes in a resolved LogicalPlan by using the nullability of corresponding Attributes of its children output Attributes.

    Updates nullability of Attributes in a resolved LogicalPlan by using the nullability of corresponding Attributes of its children output Attributes. This step is needed because users can use a resolved AttributeReference in the Dataset API and outer joins can change the nullability of an AttributeReference. Without this rule, a nullable column's nullable field can be actually set as non-nullable, which cause illegal optimization (e.g., NULL propagation) and wrong answers. See SPARK-13484 and SPARK-13801 for the concrete queries of this case.

61.  object UpdateOuterReferences extends Rule[LogicalPlan]

    The aggregate expressions from subquery referencing outer query block are pushed down to the outer query block for evaluation.

    The aggregate expressions from subquery referencing outer query block are pushed down to the outer query block for evaluation. This rule below updates such outer references as AttributeReference referring attributes from the parent/outer query block.

    For example (SQL):

    SELECT l.a FROM l GROUP BY 1 HAVING EXISTS (SELECT 1 FROM r WHERE r.d < min(l.b))

    Plan before the rule. Project [a#226] +- Filter exists#245 [min(b#227)#249] : +- Project [1 AS 1#247] : +- Filter (d#238 < min(outer(b#227))) <----- : +- SubqueryAlias r : +- Project [_1#234 AS c#237, _2#235 AS d#238] : +- LocalRelation [_1#234, _2#235] +- Aggregate [a#226], [a#226, min(b#227) AS min(b#227)#249] +- SubqueryAlias l +- Project [_1#223 AS a#226, _2#224 AS b#227] +- LocalRelation [_1#223, _2#224] Plan after the rule. Project [a#226] +- Filter exists#245 [min(b#227)#249] : +- Project [1 AS 1#247] : +- Filter (d#238 < outer(min(b#227)#249)) <----- : +- SubqueryAlias r : +- Project [_1#234 AS c#237, _2#235 AS d#238] : +- LocalRelation [_1#234, _2#235] +- Aggregate [a#226], [a#226, min(b#227) AS min(b#227)#249] +- SubqueryAlias l +- Project [_1#223 AS a#226, _2#224 AS b#227] +- LocalRelation [_1#223, _2#224]