Packages

•  package root

  Definition Classes

  root

•  package org

  Definition Classes

  root

•  package apache

  Definition Classes

  org

•  package spark

  Definition Classes

  apache

•  package sql

  Definition Classes

  spark

•  package catalyst

  Catalyst is a library for manipulating relational query plans.

  Catalyst is a library for manipulating relational query plans. All classes in catalyst are considered an internal API to Spark SQL and are subject to change between minor releases.

  Definition Classes

  sql

•  package json

  Definition Classes

  catalyst

•  class JsonFilters extends StructFilters

  The class provides API for applying pushed down source filters to rows with a struct schema parsed from JSON records.

  The class provides API for applying pushed down source filters to rows with a struct schema parsed from JSON records. The class should be used in this way: 1. Before processing of the next row, JacksonParser (parser for short) resets the internal state of JsonFilters by calling the reset() method. 2. The parser reads JSON fields one-by-one in streaming fashion. It converts an incoming field value to the desired type from the schema. After that, it sets the value to an instance of InternalRow at the position according to the schema. Order of parsed JSON fields can be different from the order in the schema. 3. Per every JSON field of the top-level JSON object, the parser calls skipRow by passing an InternalRow in which some of fields can be already set, and the position of the JSON field according to the schema. 3.1 skipRow finds a group of predicates that refers to this JSON field. 3.2 Per each predicate from the group, skipRow decrements its reference counter. 3.2.1 If predicate reference counter becomes 0, it means that all predicate attributes have been already set in the internal row, and the predicate can be applied to it. skipRow invokes the predicate for the row. 3.3 skipRow applies predicates until one of them returns false. In that case, the method returns true to the parser. 3.4 If all predicates with zero reference counter return true, the final result of the method is false which tells the parser to not skip the row. 4. If the parser gets true from JsonFilters.skipRow, it must not call the method anymore for this internal row, and should go the step 1.

  Besides of StructFilters assumptions, JsonFilters assumes that:

  • skipRow() can be called for any valid index of the struct fields, and in any order.
  • After skipRow() returns true, the internal state of JsonFilters can be inconsistent, so, skipRow() must not be called for the current row anymore without reset().

  Definition Classes

  json

• JsonPredicate

org.apache.spark.sql.catalyst.json.JsonFilters

JsonPredicate