Skip to main content

0.24.0 - 2023-05-03

Added

  • Support custom transport types #1795 @nataliezeller1
    Adds a new interface, TransportBuilder, for creating custom transport types without having to modify core components of OpenLineage.
  • Airflow: dbt Cloud integration #1418 @howardyoo
    Adds a new OpenLineage extractor for dbt Cloud that uses the dbt Cloud hook provided by Airflow to communicate with dbt Cloud via its API.
  • Spark: support dataset name modification using regex #1796 @pawel-big-lebowski
    It is a common scenario to write Spark output datasets with a location path ending with /year=2023/month=04. The Spark parameter spark.openlineage.dataset.removePath.pattern introduced here allows for removing certain elements from a path with a regex pattern.

Fixed

  • Spark: catch exception when trying to obtain details of non-existing table. #1798 @pawel-big-lebowski
    This mostly happens when getting table details on START event while the table is still not created.
  • Spark: LogicalPlanSerializer #1792 @pawel-big-lebowski
    Changes LogicalPlanSerializer to make use of non-shaded Jackson classes in order to serialize LogicalPlans. Note: class names are no longer serialized.
  • Flink: fix Flink CI #1801 @pawel-big-lebowski
    Specifies an older image version that succeeds on CI in order to fix the Flink integration.