Skip to content

Commit be0c75f

Browse files
sh-rpdjudjuuburnashVioletMtimH6502
authored
Merge devel into runtime (#3385)
* override local marimo theme for dashboard (#3337) This ensures custom CSS is always readable. * Fix DocSearch v4 styles (#3338) * Fix DocSearch v4 styles * Fix search input styles for light and dark modes * docs: update weaviate destination docs and version (#3352) * Redshift feature: Include STS session token in COPY CREDENTIALS. (#3307) * Redshift feature: Include STS session token in COPY CREDENTIALS. If aws_session_token is present, append the session token. Keeps IAM_ROLE path and long-lieved keys unchanged --------- Co-authored-by: Tim Hable <[email protected]> * fixes sqlglot from find (#3357) * fixes athena refresh mode (#3313) * adds filter to exclude dropped tables in staging destination, implements for athena * enables refresh mode tests for athena, fixes tests * fixes staging_allowed_local_path on databricks, bumps databricks connector in lockfile * passes dropped tables schemas to filter, adjust athena filter * allows to disable lake formation * fix: backwards compatible traces (#3354) * makes trace backward compat with 1.17.0 and earlier * skips trace if any error in unpickle * always saves merged pipeline trace to have consistent pipeline.last_trace property * tests for past traces, broken traces and other improvements * (docs) adds community destinations (#3326) * adds community destinations * Apply suggestions from code review applies crate fixes Co-authored-by: Andreas Motl <[email protected]> --------- Co-authored-by: Andreas Motl <[email protected]> * fix: dashboard no longer crashes on broken home cell (#3348) * split home and workspace render methods * header row dry-er * catch-all errors in home()-cell * local try-catch for broken traces * e2e test for broken trace * removes this * shows navigation on pipeline attach error --------- Co-authored-by: Marcin Rudolf <[email protected]> * (fix) use sparse checkout for dlt init dlthub (#3356) * adds option to sparse checkout repo * use sparse checkout for llm context * fixes sqlglot from find * adds checkout after sparse clone * explains unknown path tests * Fix: The child table column remains in the schema as a partial column with seen-null-first=True (#3131) * child table column removed from parent * A utility functin that checks whether a column has seen-null-first set * Improved comments and docstrings, separate method in worker * null column not inferred if exists as compound * Column level x-normalizer cleaning moved outside of worker * Test for empty column becoming compound * Test clean_seen_null_first_hint * Uncalled source in pipeline.run( (#3369) * fix flaky dashboard tests (#3370) * improves dashboard multi schema test * closes and waits for sections in multi-schema test * removes command line snippet with generic text in exceptions * disables transformers pokeapi test * feat: `Schema.to_mermaid()` (#3364) * Add dlt.Schema.to_mermaid() method --------- Co-authored-by: jayant <[email protected]> * Refactor boundary timestamp handling in SqlMergeFollowupJob and SqlalchemyMergeFollowupJob to ensure current load package creation time is used when no boundary timestamp is provided. Update DltResourceHints class to streamline timestamp validation for active_record_timestamp and boundary_timestamp. Adjust tests accordingly. (#3378) * feat: `snowflake` clustering key modifications (#3365) * add support for snowflake clustering key modifications * add cluster column order test case * update snowflake cluster hint docs * switch to reading snowflake cluster hints from table schema * docs: lifecycle of `@dlt.hub.transformation` and `dlt.Relation` (#3329) * Lifecycle of a dlt transformation * Added test to match lifecycle docs * (fix) 3351 fixes default type var (#3373) * tests minimal typing extensions in alpine docker * keeps typevar default but does not use it in the code for backwart compat --------- Co-authored-by: djudjuu <[email protected]> Co-authored-by: Anton Burnashev <[email protected]> Co-authored-by: Violetta Mishechkina <[email protected]> Co-authored-by: Tim Hable <[email protected]> Co-authored-by: Tim Hable <[email protected]> Co-authored-by: rudolfix <[email protected]> Co-authored-by: Andreas Motl <[email protected]> Co-authored-by: anuunchin <[email protected]> Co-authored-by: Thierry Jean <[email protected]> Co-authored-by: jayant <[email protected]> Co-authored-by: Menna <[email protected]> Co-authored-by: Jorrit Sandbrink <[email protected]>
1 parent b243302 commit be0c75f

File tree

75 files changed

+2770
-844
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

75 files changed

+2770
-844
lines changed

deploy/dlt/Dockerfile.minimal

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ RUN apk update &&\
1616
apk add --no-cache ca-certificates curl &&\
1717
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py && python3 get-pip.py &&\
1818
rm get-pip.py &&\
19-
pip install --upgrade setuptools wheel pip
19+
pip install --upgrade setuptools wheel pip uv
2020

2121

2222
# add build labels and envs
@@ -29,28 +29,28 @@ ENV IMAGE_VERSION=${IMAGE_VERSION}
2929

3030
# install exactly the same version of the library we used to build
3131
COPY dist/dlt-${IMAGE_VERSION}.tar.gz .
32-
RUN pip install /tmp/pydlt/dlt-${IMAGE_VERSION}.tar.gz
3332

3433
# create app dir to run simple test
3534
RUN mkdir -p /app
3635
WORKDIR /app
36+
RUN uv venv && uv pip install /tmp/pydlt/dlt-${IMAGE_VERSION}.tar.gz --resolution lowest-direct && uv pip install typing-extensions==4.8.0
3737
RUN rm -r /tmp/pydlt
3838

3939
# make sure dlt can be actually imported
40-
RUN python -c 'import dlt;import pendulum;'
40+
RUN uv run python -c 'import dlt;import pendulum;'
4141

4242
# check excluded imports
4343
COPY deploy/dlt/restrict_imports.py .
44-
RUN python restrict_imports.py
44+
RUN uv run python restrict_imports.py
4545

4646
# run simple pipeline
4747
COPY deploy/dlt/minimal_pipeline.py .
48-
RUN python minimal_pipeline.py
49-
RUN dlt pipeline fruit_pipeline info
48+
RUN uv run python minimal_pipeline.py
49+
RUN uv run dlt pipeline fruit_pipeline info
5050

5151
# enable workspace
5252
RUN mkdir -p .dlt && touch .dlt/.workspace
5353
# RUN dlt pipeline fruit_pipeline info
54-
RUN dlt workspace info
55-
RUN python minimal_pipeline.py
56-
RUN dlt pipeline fruit_pipeline info
54+
RUN uv run dlt workspace info
55+
RUN uv run python minimal_pipeline.py
56+
RUN uv run dlt pipeline fruit_pipeline info

dlt/_workspace/cli/_ai_command.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,9 @@ def vibe_source_setup(
119119
"""Copies files from vibe sources repo into the current working folder"""
120120

121121
fmt.echo("Looking up in dltHub for rules, docs and snippets for %s..." % fmt.bold(source))
122-
src_storage = git.get_fresh_repo_files(location, get_dlt_repos_dir(), branch=branch)
122+
src_storage = git.get_fresh_repo_files(
123+
location, get_dlt_repos_dir(), branch=branch, path=source
124+
)
123125
if not src_storage.has_folder(source):
124126
fmt.warning("We have nothing for %s at dltHub yet." % fmt.bold(source))
125127
return

dlt/_workspace/cli/_pipeline_command.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -378,6 +378,8 @@ def _display_pending_packages() -> Tuple[Sequence[str], Sequence[str]]:
378378
schema_str = s.to_dbml()
379379
elif format_ == "dot":
380380
schema_str = s.to_dot()
381+
elif format_ == "mermaid":
382+
schema_str = s.to_mermaid()
381383
else:
382384
schema_str = s.to_pretty_yaml(remove_defaults=remove_defaults_)
383385

dlt/_workspace/cli/commands.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -488,6 +488,8 @@ def schema_command_wrapper(file_path: str, format_: str, remove_defaults: bool)
488488
schema_str = s.to_dbml()
489489
elif format_ == "dot":
490490
schema_str = s.to_dot()
491+
elif format == "mermaid":
492+
schema_str = s.to_mermaid()
491493
else:
492494
schema_str = s.to_pretty_yaml(remove_defaults=remove_defaults)
493495

0 commit comments

Comments
 (0)