0.2.5

What's Changed

• feat: Add calibration support to map operations for improved consistency by @shreyashankar in #365
• chore: add support for docling-serve v1alpha API with legacy fallback by @shreyashankar in #364
• fix: create join optimizer object correctly by @shreyashankar in #366
• fix: reduce optimizer bugs by @shreyashankar in #368
• fix: move truncate messages before adding the system prompt by @shreyashankar in #369
• docs: improve gleaning description by @shreyashankar in #370
• docs: improve gleaning description by @shreyashankar in #371
• fix: improve caching and don't raise error for bad gather configs by @shreyashankar in #373
• feat: adding conditional gleaning by @shreyashankar in #375
• chore: bump up fastapi and python multipart by @shreyashankar in #376
• Clean up and reorganize pytest tests by @shreyashankar in #377
• Refactor api.py for structured output by @shreyashankar in #378
• Add operators to pandas API by @shreyashankar in #379
• feat: add global bypass cache by @shreyashankar in #383
• Update cluster docs to use jinja loops by @shreyashankar in #386
• fix: truncate documents for embeddings in resolve by @shreyashankar in #385
• Fix github issue 384 by @shreyashankar in #387
• Have make lint pass by @sidjha1 in #388
• Add stubs for PyYAML and requests by @sidjha1 in #390
• Remove old typing imports by @sidjha1 in #389
• Clean up utility code by @sidjha1 in #391
• Improve syntax_check by leveraging pydantic validation by @sidjha1 in #392
• Add structured output mode support to pandas API by @shreyashankar in #394
• add new demo by @shreyashankar in #395
• website: add SEO for showcase by @shreyashankar in #396
• website: improving SEO by @shreyashankar in #397
• website: add seo by @shreyashankar in #398
• add SEO by @shreyashankar in #399
• chore: switching to cloudbank for blob storage by @shreyashankar in #400
• Switch from poetry to uv by @shreyashankar in #402

New Contributors

• @sidjha1 made their first contribution in #388

Full Changelog: 0.2.4...0.2.5

0.2.4

What's Changed

• fix: add system prompt and other config vars to python api by @shreyashankar in #356
• Support pandas dataframe as input by @sfc-gh-jdu in #357
• feat: adding api_base to yaml by @shreyashankar in #359
• feat: add extract operator by @shreyashankar in #361
• Update README.md by @sotoblanco in #358

New Contributors

• @sotoblanco made their first contribution in #358

Full Changelog: 0.2.3...0.2.4

0.2.3

What's Changed

• expts: assess performance of structured outputs by @shreyashankar in #291
• Handle DeepSeek R1 distil models: no tool calling and add think column by @lemig in #295
• chore: update packages by @shreyashankar in #300
• Allow import of text files in non utf-8 encoding by @lemig in #299
• Detect case variations: DeepSeek-R1 vs deepseek-r1 by @lemig in #304
• 🚀 feat: add docker-compose support by @Sunwood-ai-labs in #301
• feat: allow optimizer model to be any litellm supported model by @shreyashankar in #307
• feat: support native pdf processing capabilities by gemini and claude LLMs by @shreyashankar in #309
• feat: refactoring map optimizer by @shreyashankar in #311
• feat: add rate limiting in optimizer by @shreyashankar in #313
• feat: add NL to pipeline dialog by @shreyashankar in #314
• fix: warning in pd_accessors.py by @shreyashankar in #316
• feat: add PDF url key option to DocWrangler for Map operation by @shabie in #315
• fix: uploading pipeline yaml with complex schemas by @shreyashankar in #317
• feat: add n parameter to output by @shreyashankar in #320
• Update sample.py by @shreyashankar in #322
• feat: add flag to stream map operation outputs to disk by @shabie in #323
• perf: remove client side JSON validation logic by @shreyashankar in #325
• fix: don't print locals in rich traceback by @shreyashankar in #329
• feat: Render vega or vega-lite code blocks as interactive graphs by @hydrosquall in #327
• feat: add gpt tokenizer to webisite by @shreyashankar in #330
• feat: add AWS support by @shabie in #331
• docs: Add python doc examples. by @shreyashankar in #334
• feat: add _file_path to each document when uploading the dataset by @shreyashankar in #335
• feat: Add support for Snowflake Cortex as an LLM provider by @sfc-gh-jdu in #336
• docs: add python api quickstart by @shreyashankar in #337
• feat: add TPM rate limits & pipeline settings to DocWrangler by @shreyashankar in #338
• feat: add rank operation by @shreyashankar in #340
• feat: add rank operation card in docwrangler by @shreyashankar in #341
• docs: update documentation for rank op by @shreyashankar in #343
• docs: update for rank op by @shreyashankar in #344
• fix: get rid of namespace infinite loop by @shreyashankar in #345
• nit: update pricing calculation for docwrangler assistant by @shreyashankar in #346
• Update rank.py by @shreyashankar in #347
• feat: add a demo for AI RFI Response Analysis by @shreyashankar in #348
• nit: add disclaimer to demo by @shreyashankar in #349
• Use hasattr when checking response.choices[index].message by @sfc-gh-jdu in #350
• feat: add prompt example showcase by @shreyashankar in #351
• feat: adding hosted llm and azure DI to DW by @shreyashankar in #352
• Add "allowed_openai_params" argument for Snowflake model by @sfc-gh-jdu in #353
• bump version by @shreyashankar in #354

New Contributors

• @lemig made their first contribution in #295
• @Sunwood-ai-labs made their first contribution in #301
• @shabie made their first contribution in #315
• @hydrosquall made their first contribution in #327
• @sfc-gh-jdu made their first contribution in #336

Full Changelog: 0.2.2...0.2.3

0.2.2

What's Changed

• refactor: DSLRunner now uses a pull-based execution model by @shreyashankar in #273
• feat: add new pipeline of customer support tickets by @shreyashankar in #276
• tests: fix makefile by @shreyashankar in #277
• fix: performance optimization for dataset loading by @shreyashankar in #279
• feat: quality of life improvements for equijoin optimizer by @shreyashankar in #280
• docs: add verbose param by @shreyashankar in #283
• fix: allow litellm completion kwargs to work in the FE by @shreyashankar in #285
• feat: add remote upload ability and csv upload ability by @shreyashankar in #289
• feat: add pandas df accessor by @shreyashankar in #287
• feat: add enum to UI by @shreyashankar in #292
• fix: input context length logic by @shreyashankar in #293

Full Changelog: 0.2.1...0.2.2

0.2.1

What's Changed

• fix: make sure show outputs works after namespaces by @shreyashankar in #230
• chore: update lockfile by @shreyashankar in #231
• local GPU's conversion of PDFs is really too slow. Just can't bear it… by @plpycoin in #229
• feat: clean up UI to look a bit more consistent and smooth by @shreyashankar in #233
• fix: adding sampling back by @shreyashankar in #236
• fix: add strict jinja templating by @shreyashankar in #237
• fix: solve infinite rerender bug by @shreyashankar in #241
• WIP: separate frontend from backend so we can host the frontend by @shreyashankar in #242
• fix: adding openai key in request header for chatbot in playground by @shreyashankar in #243
• chore: optimize website for mobile by @shreyashankar in #244
• feat: improve website layout by @shreyashankar in #245
• fix: website resizing was causing problems with demo by @shreyashankar in #246
• fix: show namespace dialog on load by @shreyashankar in #247
• UI nits by @shreyashankar in #249
• feat: add sample pipeline for supreme court hearing transcripts by @shreyashankar in #250
• fix: output csv writing should work even if not all documents have all keys by @shreyashankar in #252
• added enum support by @rrawatt in #254
• chore: add param to skip LLM calls when they fail by @shreyashankar in #255
• feat: add azure openai for the FE assistants by @shreyashankar in #256
• refactor: updating UI text to docwrangler by @shreyashankar in #257
• fix: map optimizer metadata error by @shreyashankar in #258
• docs: update playground docs by @shreyashankar in #259
• chore: better logging by @shreyashankar in #262
• feat: add ability for user to specify their docling server by @shreyashankar in #263
• fix: bypass vercel serverless functions when uploading datasets by @shreyashankar in #266
• feat: add llmstxt by @shreyashankar in #267
• fix: equijoin is out of date by @shreyashankar in #269
• chore: adding better error messages for gemini by @shreyashankar in #271

New Contributors

• @rrawatt made their first contribution in #254

Full Changelog: 0.2...0.2.1

0.2.0

What's Changed

• Sample by @redhog in #92
• Outliers by @redhog in #91
• Sample (+ Outlier Functionality) Operation by @shreyashankar in #100
• #91 document > item renaming by @garuna-m6 in #103
• chore: update dependency versions by @shreyashankar in #105
• docs: add 'output' argument to ResolveOp code eg by @goutham794 in #106
• fix: edit agent for synth resolve task by @shreyashankar in #109
• fix: update python api with cluster & sample ops by @shreyashankar in #113
• docs: add sample and cluster to docs by @shreyashankar in #114
• New api by @redhog in #115
• fix: make docs work by @shreyashankar in #119
• feat: adding human in the loop for split-map-gather decomp by @shreyashankar in #120
• fix: cache partial pipeline runs by @shreyashankar in #122
• mark as flaky test by @shreyashankar in #123
• only compare distinct pairs in resolve by @shreyashankar in #124
• LinkResolveOperation by @redhog in #117
• Fix Resolve and Map progress bars by @michielree in #126
• Better auto batching for resolve LLM calls by @sushruth2003 in #128
• Merge auto batching PR by @shreyashankar in #129
• v1 of the UI! by @shreyashankar in #118
• Upgrade litellm version to v1.51.0-stable by @Tendo33 in #131
• feat: adding batching for map and filter calls by @shreyashankar in #133
• docs: link filter to map by @shreyashankar in #135
• fix: optimizer bug where the reduce operation can't be optimized with azure by @shreyashankar in #136
• fix: only call os makedirs on non empty paths by @shreyashankar in #137
• UI: add basic chat-based assistant by @shreyashankar in #139
• fix: clear and run button should also bypass cache by @shreyashankar in #140
• feat: UDFs support added by @staru09 in #138
• Merge staging into MAIN by @shreyashankar in #141
• chore, load envs from current directory by @plpycoin in #142
• feat: add optimizer in the UI by @shreyashankar in #143
• chore: Use environment variable configuration files to set host, port by @plpycoin in #146
• fix: render cells in markdown and fix resizable panels by @shreyashankar in #148
• hotfix: an error occurred when running make run-ui by @plpycoin in #152
• fix: switch crypto uuid to regular uuid by @shreyashankar in #155
• feat: provide defaults in the UI chat by @shreyashankar in #156
• fix: allow reduce_key types to be lists by @shreyashankar in #162
• fix: allow user to pass in litellm completion kwargs by @shreyashankar in #163
• Sagemaker doesn't yet support tools by @njbrake in #165
• fix: save validation and gleaning by @shreyashankar in #167
• Remove unnecessary console log statements by @shreyashankar in #168
• chore: update docs to link to paper by @shreyashankar in #174
• feature: add automatic optimization check to the UI (opt in) by @shreyashankar in #175
• fix: ts errors by @shreyashankar in #176
• fix: ts errors by @shreyashankar in #177
• feat: tie histograms to output types by @shreyashankar in #178
• Change for issue: #180 by @yogitha2023 in #181
• make test less flaky by @shreyashankar in #182
• feat: add code operations to the ui (#169) by @shreyashankar in #183
• hotfix: Import declaration conflicts with local declaration of 'Operation'. by @plpycoin in #185
• chore: make output visualizations better by @shreyashankar in #186
• chore: edit copy for the UI by @shreyashankar in #189
• feat: add pdf upload for the UI by @shreyashankar in #190
• fix sampling in second op onwards by @shreyashankar in #192
• Receive a customized parameter to specify the number of concurrent running threads for the code_* operations. by @plpycoin in #195
• chore: fix optimizer and API for user study by @shreyashankar in #196
• add python multipart to requirements by @samelamin in #197
• fix: add docs to describe how to set up .env.local by @shreyashankar in #199
• Intermediates fix by @samelamin in #200
• fix: change default settings by @shreyashankar in #202
• feat: Add azure doc intelligence to PDF upload in the UI by @shreyashankar in #203
• feat: change histograms to be bar charts for categorical columns by @shreyashankar in #204
• ensure the value is converted to a string when it's not an object by @plpycoin in #206
• feat: add docker file by @shreyashankar in #205
• feat: add basic llm call observability to the UI by @shreyashankar in #209
• feat: have global system prompt and decription by @shreyashankar in #210
• fix: small errors in user study by @shreyashankar in #211
• fix: support markdown files by @shreyashankar in #213
• Add variable descriptions that limit the number of concurrent threads by @plpycoin in #215
• feat: add column view dialog by @shreyashankar in #214
• Update reduce folding instruction to be clearer by @shreyashankar in #217
• fix: make histogram calculation and rendering less blocking by @shreyashankar in #218
• edit system prompt in prompt improvement by @shreyashankar in #223
• fix: edit system prompt for prompt rewriter by @shreyashankar in #224
• Fix cache naming by @sushruth2003 in #220
• refactor recursive optimization for map operations by @shreyashankar in #225
• feat: adding namespaces by @shreyashankar in #226

New Contributors

• @garuna-m6 made their first contribution in #103
• @goutham794 made their first contribution in #106
• @michielree made their first contribution in #126
• @sushruth2003 made their first contribution in #128
• @Tendo33 made their first contribution in #131
• @plpycoin made their first contribution in #142
• @njbrake made their first contribution in #165
• @yogitha2023 made their first contribution in #181
• @samelamin made their first contribution in #197

Full Changelog: 0.1.7...0.2.0

0.1.7

What's Changed

• Add Operation Hash and Caching Functionality by @shreyashankar in #61
• docs: improving documentation for pipeline api by @shreyashankar in #62
• refactor: adding website code by @shreyashankar in #65
• (partial) fix: add exponential backoff for rate limit errors by @shreyashankar in #66
• fix: enable gleaning llm calls to work by @shreyashankar in #70
• Added llama-index based parsers by @redhog in #71
• Merging staging to main by @shreyashankar in #74
• feat: add pdfgpt to parse PDFs by @staru09 in #67
• Merging staging to main (from add gpt_pdf) by @shreyashankar in #76
• fix: disable additional properties for gemini by @shreyashankar in #73
• Throttle by @redhog in #64
• feat: support rate limits by @shreyashankar in #79
• feat: add verbose parameter for gleaning by @shreyashankar in #80
• Parsers can now return any number of fields, and can access the whole item by @redhog in #81
• Merge staging to main (after parsers refactor) by @shreyashankar in #82
• docs: add sample parameter by @shreyashankar in #87
• Clustering by @redhog in #84
• Merge staging to main (after adding cluster operator) by @shreyashankar in #88
• feat: output to csv if user specifies a csv file by @shreyashankar in #89
• Rename internal methods by @redhog in #90
• Nits for cleaning up API. by @shreyashankar in #93
• refactor: move validation and gleaning into call llm by @shreyashankar in #98
• Staging to main by @shreyashankar in #99
• feat: add reduce operation lineage by @shreyashankar in #101
• fix: change gleaning prompt to validation_prompt by @shreyashankar in #102

New Contributors

• @staru09 made their first contribution in #67

Full Changelog: 0.1.6...0.1.7

0.1.6

What's Changed

• docs: fix resolve docs by @shreyashankar in #27
• docs: link to ollama chat by @shreyashankar in #28
• feat: show better progress bars for operations by @shreyashankar in #30
• Add batching support to map operations with configurable parameters by @orban in #16
• feat: implement batch limit in map operations by @shreyashankar in #31
• Add Dataset Class and Parsing Tools by @shreyashankar in #32
• docs: improve clarity for custom parsing by @shreyashankar in #34
• Add Azure Document Intelligence Read Tool by @shreyashankar in #36
• fix: read .env from the user's cwd and change tool schema so ollama llama models work better by @shreyashankar in #38
• Bugfix for sqlite3 operation error in cache by @redhog in #40
• fix: make diskcache reads thread-safe by @shreyashankar in #42
• fix: template in tutorial.md by @shreyashankar in #43
• RateLimit error by @redhog in #39
• feat: add paddleocr by @shreyashankar in #44
• Entrypoints by @redhog in #45
• fix: dont cache results of bad llm calls by @shreyashankar in #52
• fix: default to gpt 4o tokenizer by @shreyashankar in #57
• feat: print out LLM message history and tools when there's an InvalidOutputError by @shreyashankar in #53
• feat: don't use tool calling for ollama/OSS models if the output schema is just one param by @shreyashankar in #59
• docs: add documentation for using split gather pipeline by @shreyashankar in #60

New Contributors

• @orban made their first contribution in #16
• @redhog made their first contribution in #40

Full Changelog: 0.1.5...0.1.6

0.1.5

What's Changed

• fix: add error messages if model doesnt support tool calling by @shreyashankar in #26

Full Changelog: 0.1.4...0.1.5

v0.1.4

What's Changed

• fix: manually try to parse ollama outputs, even if it is not valid json by @shreyashankar in #25

Full Changelog: 0.1.3...0.1.4