Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
7b4086a
Fix field type from TIME to DATETIME in BigQuery converter and schema
jochenchrist Apr 16, 2025
9f62ad7
Update dependency constraints for python-dotenv and boto3
simonharrer Apr 16, 2025
3bc0b9e
Update pre-commit requirement from <4.2.0,>=3.7.1 to >=3.7.1,<4.3.0 (…
dependabot[bot] Apr 18, 2025
8891807
Feature/odcsv3 #715 (#724)
ezhao-mck Apr 18, 2025
4a133c1
Update databricks-sql-connector requirement (#721)
dependabot[bot] Apr 18, 2025
02eb4b1
Bump moto from 5.1.1 to 5.1.3 (#720)
dependabot[bot] Apr 18, 2025
e3a584a
Update dependency constraints for various packages to allow newer ver…
jochenchrist Apr 19, 2025
944d48b
Update changelog
jochenchrist Apr 19, 2025
48ec80b
feat: more descriptive csv importer (#692)
stefanedwards Apr 19, 2025
4b3020d
Deprecated QualityLinter is now removed
jochenchrist Apr 19, 2025
ce95fa5
chore: remove clevercsv dependency from pyproject.toml
jochenchrist Apr 19, 2025
a44b0f5
deep nesting of json objects in duckdb (#681)
stefanedwards Apr 19, 2025
388deb3
chore: update changelog to include datacontract test with DuckDB
jochenchrist Apr 19, 2025
91d8dfe
chore: update version to 0.10.24
jochenchrist Apr 19, 2025
8af13e1
Resolve FastAPI deprecation warning for example fields (#730)
emmanuel-ferdman Apr 19, 2025
50ce8a5
Update databricks-sdk requirement from <0.50.0 to <0.51.0 (#733)
dependabot[bot] Apr 24, 2025
d8943af
In export to odcs, keep nested structures in the schema (currently fl…
gilles-guglielmoni Apr 29, 2025
a6ce1f6
Include custom quality rules in great expectations export (#738)
sauger92 May 5, 2025
869a578
fix: update help text for source option in cli.py
jochenchrist May 5, 2025
33165b8
Use pip package for dcs (#705)
simonharrer May 6, 2025
6dc0a7d
Use Open Data Contract Standard pip module (#746)
simonharrer May 7, 2025
fbf4d0b
Add example for testing with local files (#743)
mitchtech May 7, 2025
33e238a
chore: update changelog for version 0.10.25 and bump version in pypro…
jochenchrist May 7, 2025
718e47b
UPDATE
simonharrer May 7, 2025
7c561ba
fix: synchronize pyspark version in kafka.py with pyproject.toml
simonharrer May 7, 2025
5172d32
fix: correct string formatting in breaking_change.py
simonharrer May 8, 2025
f219ef5
fix: correct string formatting in breaking_change.py
simonharrer May 8, 2025
23b44a8
Bump pymssql from 2.3.2 to 2.3.4 (#734)
dependabot[bot] May 8, 2025
407f7ce
Bump uvicorn from 0.34.0 to 0.34.2 (#735)
dependabot[bot] May 8, 2025
22611af
fix: update Databricks environment variable names in README.md
simonharrer May 14, 2025
f9200d4
ODCS export: Export physical type if the physical type is configured …
jochenchrist May 15, 2025
cc12bd9
Include descriptions from the data contract when exporting to snowfla…
himat-mesh-ai May 16, 2025
19d4e3d
Add support for Variant type (#758)
robert-altmiller May 16, 2025
538f533
chore: update changelog for version 0.10.26 and bump version in pypro…
jochenchrist May 16, 2025
79d752c
Update rich requirement from <14.0,>=13.7 to >=13.7,<15.0 (#731)
dependabot[bot] May 17, 2025
3ffeb16
Complex Types Fix When Exporting to SQL and 'databricksType' is Defin…
robert-altmiller May 17, 2025
bf79ee3
Update pydantic requirement from <2.11.0,>=2.8.2 to >=2.8.2,<2.12.0 (…
dependabot[bot] May 17, 2025
ab03764
Update aiobotocore requirement from <2.20.0,>=2.17.0 to >=2.17.0,<2.2…
dependabot[bot] May 19, 2025
882e209
Bump testcontainers[kafka,minio,mssql,postgres] from 4.9.2 to 4.10.0 …
dependabot[bot] May 19, 2025
8f349d9
Bump moto from 5.1.3 to 5.1.4 (#765)
dependabot[bot] May 19, 2025
447e86e
add: s3 dependencies allow upward migration
simonharrer May 19, 2025
7257137
Added support for spark importer table level comments (#761)
robert-altmiller May 22, 2025
07fbf27
chore: update changelog to include spark importer table level comments
jochenchrist May 22, 2025
330b092
fix: Soda Exporter did not correctly pass server object to create_che…
Christopher-Lawford May 22, 2025
ced6aa0
feat: add owner and id options to data contract import functionality …
jochenchrist May 22, 2025
edf8bb8
Change behaviour of DBT constraints vs data tests when no model type …
skidave May 22, 2025
fc9a0bf
Adding mermaid format for export and adding mermaid figure in html ex…
mathispernias May 22, 2025
0f0493a
fix: change return type of export methods in SqlExporter and SqlQuery…
jochenchrist May 22, 2025
4b6b9a1
chore: update changelog to reflect changes for mermaid export and spa…
jochenchrist May 22, 2025
fa3db82
Bump version
jochenchrist May 22, 2025
52ad956
chore: add pyspark dependency version 3.5.5 to pyproject.toml
jochenchrist May 27, 2025
5fd1e83
Added support to spark importer to be able to import Spark dataframe …
robert-altmiller May 27, 2025
f63d27d
chore: remove pyspark dependency version constraint from pyproject.toml
jochenchrist May 27, 2025
d876d84
chore(deps): update databricks-sdk requirement from <0.51.0 to <0.55.…
dependabot[bot] May 27, 2025
b2992e0
chore(deps): bump moto from 5.1.4 to 5.1.5 (#772)
dependabot[bot] May 27, 2025
2224770
Add data contract version number to index.html (#780)
vtulus Jun 3, 2025
7280d8e
Better odcs support (#783)
simonharrer Jun 5, 2025
43ab0f5
Release 0.10.28
simonharrer Jun 5, 2025
7ff3510
Release 0.10.28
simonharrer Jun 5, 2025
0dc8b61
Fix Error importing roles: list index out of range
jochenchrist Jun 5, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
75 changes: 74 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,80 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- Fix to handle logicalType format wrt avro mentioned in issue #687
## [0.10.28] - 2025-06-05

### Added
- Much better ODCS support
- Import anything to ODCS via the `import --spec odcs` flag
- Export to HTML with an ODCS native template via `export --format html`
- Export to Mermaid with an ODCS native mapping via `export --format mermaid`
- The databricks `unity` importer now supports more than a single table. You can use `--unity-table-full-name` multiple times to import multiple tables. And it will automatically add a server with the catalog and schema name.

### Changed
- `datacontract catalog [OPTIONS]`: Added version to contract cards in `index.html` of the catalog (enabled search by version)
- The type mapping of the `unity` importer no uses the native databricks types instead of relying on spark types. This allows for better type mapping and more accurate data contracts.

### Fixed

## [0.10.27] - 2025-05-22

### Added

- `datacontract export --format mermaid` Export
to [Mermaid](https://mermaid-js.github.io/mermaid/#/) (#767, #725)

### Changed

- `datacontract export --format html`: Adding the mermaid figure to the html export
- `datacontract export --format odcs`: Export physical type to ODCS if the physical type is
configured in config object
- `datacontract import --format spark`: Added support for spark importer table level comments (#761)
- `datacontract import` respects `--owner` and `--id` flags (#753)

### Fixed

- `datacontract export --format sodacl`: Fix resolving server when using `--server` flag (#768)
- `datacontract export --format dbt`: Fixed DBT export behaviour of constraints to default to data tests when no model type is specified in the datacontract model


## [0.10.26] - 2025-05-16

### Changed
- Databricks: Add support for Variant type (#758)
- `datacontract export --format odcs`: Export physical type if the physical type is configured in
config object (#757)
- `datacontract export --format sql` Include datacontract descriptions in the Snowflake sql export (
#756)

## [0.10.25] - 2025-05-07

### Added
- Extracted the DataContractSpecification and the OpenDataContractSpecification in separate pip modules and use them in the CLI.
- `datacontract import --format excel`: Import from Excel
template https://github.com/datacontract/open-data-contract-standard-excel-template (#742)

## [0.10.24] - 2025-04-19

### Added

- `datacontract test` with DuckDB: Deep nesting of json objects in duckdb (#681)

### Changed

- `datacontract import --format csv` produces more descriptive output. Replaced
using clevercsv with duckdb for loading and sniffing csv file.
- Updated dependencies

### Fixed

- Fix to handle logicalType format wrt avro mentioned in issue (#687)
- Fix field type from TIME to DATETIME in BigQuery converter and schema (#728)
- Fix encoding issues. (#712)
- ODCS: Fix required in export and added item and fields format (#724)

### Removed

- Deprecated QualityLinter is now removed

## [0.10.23] - 2025-03-03

Expand Down
786 changes: 402 additions & 384 deletions README.md

Large diffs are not rendered by default.

6 changes: 3 additions & 3 deletions datacontract/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ async def test(
server: Annotated[
str | None,
Query(
example="production",
examples=["production"],
description="The server name to test. Optional, if there is only one server.",
),
] = None,
Expand Down Expand Up @@ -191,7 +191,7 @@ async def lint(
schema: Annotated[
str | None,
Query(
example="https://datacontract.com/datacontract.schema.json",
examples=["https://datacontract.com/datacontract.schema.json"],
description="The schema to use for validation. This must be a URL.",
),
] = None,
Expand Down Expand Up @@ -220,7 +220,7 @@ def export(
server: Annotated[
str | None,
Query(
example="production",
examples=["production"],
description="The server name to export. Optional, if there is only one server.",
),
] = None,
Expand Down
2 changes: 1 addition & 1 deletion datacontract/catalog/catalog.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
from jinja2 import Environment, PackageLoader, select_autoescape

from datacontract.data_contract import DataContract
from datacontract.export.html_export import get_version
from datacontract.export.html_exporter import get_version
from datacontract.model.data_contract_specification import DataContractSpecification


Expand Down
25 changes: 21 additions & 4 deletions datacontract/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

from datacontract.catalog.catalog import create_data_contract_html, create_index_html
from datacontract.data_contract import DataContract, ExportFormat
from datacontract.imports.importer import ImportFormat
from datacontract.imports.importer import ImportFormat, Spec
from datacontract.init.init_template import get_init_template
from datacontract.integration.datamesh_manager import (
publish_data_contract_to_datamesh_manager,
Expand Down Expand Up @@ -126,7 +126,8 @@ def test(
"servers (default)."
),
] = "all",
publish: Annotated[str, typer.Option(help="The url to publish the results after the test")] = None,
publish_test_results: Annotated[bool, typer.Option(help="Publish the results after the test")] = False,
publish: Annotated[str, typer.Option(help="DEPRECATED. The url to publish the results after the test.")] = None,
output: Annotated[
Path,
typer.Option(
Expand All @@ -149,6 +150,7 @@ def test(
run = DataContract(
data_contract_file=location,
schema_location=schema,
publish_test_results=publish_test_results,
publish_url=publish,
server=server,
ssl_verification=ssl_verification,
Expand Down Expand Up @@ -244,8 +246,12 @@ def import_(
] = None,
source: Annotated[
Optional[str],
typer.Option(help="The path to the file or Glue Database that should be imported."),
typer.Option(help="The path to the file that should be imported."),
] = None,
spec: Annotated[
Spec,
typer.Option(help="The format of the data contract to import. "),
] = Spec.datacontract_specification,
dialect: Annotated[
Optional[str],
typer.Option(help="The SQL dialect to use when importing SQL files, e.g., postgres, tsql, bigquery."),
Expand All @@ -265,7 +271,7 @@ def import_(
),
] = None,
unity_table_full_name: Annotated[
Optional[str], typer.Option(help="Full name of a table in the unity catalog")
Optional[List[str]], typer.Option(help="Full name of a table in the unity catalog")
] = None,
dbt_model: Annotated[
Optional[List[str]],
Expand Down Expand Up @@ -297,13 +303,22 @@ def import_(
str,
typer.Option(help="The location (url or path) of the Data Contract Specification JSON Schema"),
] = None,
owner: Annotated[
Optional[str],
typer.Option(help="The owner or team responsible for managing the data contract."),
] = None,
id: Annotated[
Optional[str],
typer.Option(help="The identifier for the the data contract."),
] = None,
):
"""
Create a data contract from the given source location. Saves to file specified by `output` option if present, otherwise prints to stdout.
"""
result = DataContract().import_from_source(
format=format,
source=source,
spec=spec,
template=template,
schema=schema,
dialect=dialect,
Expand All @@ -316,6 +331,8 @@ def import_(
dbml_schema=dbml_schema,
dbml_table=dbml_table,
iceberg_table=iceberg_table,
owner=owner,
id=id,
)
if output is None:
console.print(result.to_yaml(), markup=False, soft_wrap=True)
Expand Down
Loading
Loading