Skip to main content

1.9.1 - 2024-02-26

info

This version adds the capability to publish Scala 2.12 and 2.13 variants of Apache Spark, which necessitates a change in the artifact identifier for io.openlineage:openlineage-spark. From this version onwards, please use:
io.openlineage:openlineage-spark_${SCALA_BINARY_VERSION}:${OPENLINEAGE_SPARK_VERSION}.

Added​

  • Airflow: add support for JobTypeJobFacet properties #2412 @mattiabertorello
    Adds support for Job type properties within the Airflow Job facet.
  • dbt: add support for JobTypeJobFacet properties #2411 @mattiabertorello
    Support Job type properties within the DBT Job facet.
  • Flink: support Flink Kafka dynamic source and sink #2417 @HuangZhenQiu
    Adds support for Flink Kafka Table Connector use cases for topic and schema extraction.
  • Flink: support multi-topic Kafka Sink #2372 @pawel-big-lebowski
    Adds support for multi-topic Kafka sinks. Limitations: recordSerializer needs to implement KafkaTopicsDescriptor. Please refer to the limitations sections in documentation.
  • Flink: support lineage for JDBC connector #2436 @HuangZhenQiu
    Adds support for use cases that employ this connector.
  • Flink: add common config gradle plugin #2461 @HuangZhenQiu
    Add common config gradle plugin to simplify gradle files of Flink submodules.
  • Java: extend circuit breaker loaded with ServiceLoader #2435 @pawel-big-lebowski
    Loads the circuit breaker builder with ServiceLoader as an addition to a list of implemented builders available within the existing package.
  • Spark: integration now emits intermediate, application level events wrapping entire job execution #2371 @mobuchowski
    Previously, the Spark event model described only single actions, potentially linked only to some parent run. Closes #1672.
  • Spark: support built-in lineage within DataSourceV2Relation #2394 @pawel-big-lebowski
    Enables built-in lineage extraction within from DataSourceV2Relation lineage nodes.
  • Spark: add support for JobTypeJobFacet properties #2410 @mattiabertorello
    Adds support for Job type properties within the Spark Job facet.
  • Spark: stop sending spark.LogicalPlan facet by default #2433 @pawel-big-lebowski
    spark.LogicalPlan has been added to default value of spark.openlineage.facets.disabled.
  • Spark/Flink/Java: circuit breaker #2407 @pawel-big-lebowski
    Introduces a circuit breaker mechanism to prevent effects of over-instrumentation. Implemented within Java client, it serves both the Flink and Spark integration. Read the Java client README for more details.
  • Spark: add the capability to publish Scala 2.12 and 2.13 variants of openlineage-spark #2446 @d-m-h
    Adds the capability to publish Scala 2.12 and 2.13 variants of openlineage-spark

Changed​

  • Spark: enable the app module to be compiled with Scala 2.12 and Scala 2.13 variants of Apache Spark #2432 @d-m-h
    The spark.binary.version and spark.version properties control which variant to build.
  • Spark: enable Scala 2.13 support in the app module #2432 @d-m-h
    Enables the app module to be built using both Scala 2.12 and Scala 2.13 variants of various Apache Spark versions, and enables the CI/CD pipeline to build and test them.
  • Spark: don't fail on exception of UnknownEntryFacet creation #2431 @mobuchowski
    Failure to generate UnknownEntryFacet was resulting in the event not being sent.
  • Spark: move Snowflake code into the vendor projects folders #2405 @mattiabertorello
    Creates a vendor folder to isolate Snowflake-specific code from the main Spark integration, enhancing organization and flexibility.

Fixed​

  • Flink: resolve PMD rule violation warnings #2403 @HuangZhenQiu
    Resolves the PMD rule violation warnings in the Flink integration module.
  • Flink: Added the 'isReleaseVersion' property back to the build, enabling the Flink integration to be release #2468 @d-m-h
    The 'isReleaseVersion' property was removed from the build, preventing the Flink integration from being released.
  • Python: fix issue with file config creating additional file #2447 @kacpermuda
    FileConfig was creating an additional file when not in append mode. Closes #2439.
  • Python: fix issue with append option in file config #2441 @kacpermuda
    FileConfig was ignoring the append key in YAML config. Closes #2440
  • Spark: fix integration catalog symlink without warehouse #2379 @algorithmy1
    In the case of symlinked Glue Catalog Tables, the parsing method was producing dataset names identical to the namespace.
  • Flink: fix IcebergSourceWrapper for Iceberg connector 1.17 #2409 @ensctom
    In Flink 1.17, the Iceberg catalogloader was loading the catalog in the open function, causing the loadTable method to throw a NullPointerException error.
  • Spark: migrate spark35, spark3, shared modules to produce Scala 2.12 and Scala 2.13 variants #2390 #2385#2384 @d-m-h
    Migrates the three modules to use the refactored Gradle plugins. Also splits some tests into Scala 2.12- and Scala 2.13-specific versions.
  • Spark: conform the spark2 module to the new build process #2391 @d-m-h
    Due to a change in the Scala Collections API in Scala 2.13, NoSuchMethodErrors were being thrown when running the openlineage-spack connector in an Apache Spark runtime compiled using Scala 2.13.