WEEKLY OSS REVIEW

A running log of weekly open-source progress, reviews, and merged work

Week of Sep 29 – Oct 4, 2026

Status: draft · fill in this week's notes

Add this week's open-source progress here: PRs opened/merged, reviews given, issues triaged, and anything learned.

#1 Missing BOS token made a healthy base model generate garbageunslothai/unsloth#10312

A missing special token didn't crash the model.It made a perfectly valid base LLM generate garbage , and the bug only became obvious when we compared the exact input IDs.

My model was loaded correctly. Prompt was correctThe seed was fixed.And Gemma 4 E2B base still repeated the same phrase forever:

This book is largely concerned with Hobbits, and from its pages,“its pages, its pages, its pages, its page”

Then I made one change:Add a <bos> token to the beginning of the input.Same model.Same prompt.Same seed.The output became coherent.

The model itself wasn't broken.The problem was in the input-tokenization path.Gemma 4 base models could reach inference without the BOS behavior expected for raw completion. The -it models didn't show the same problem because their chat templates already emit BOS.

That made this easy to misdiagnose:It looked like a model-generation problem.It was actually an input-tokenization problem.

The first fix was straightforward: enable BOS for the affected Gemma 4 base models.But review uncovered more edge cases.

A simple model-name check wasn't enough:→ Local paths don't necessarily match the Hub name→ Quantized models can have additional suffixes→ Chat templates may already emit BOS→ Adding another one creates duplicate BOS tokens

So the final implementation detects Gemma 4 from the loaded tokenizer/model configuration, rather than relying only on the model name.For raw completion, it repairs the missing BOS behavior.For chat, it checks whether the rendered prompt already contains BOS before adding another.

The final verification was more interesting than simply checking whether the test passed.For unsloth/gemma-4-E2B, with seed 3407:Before the fix:49–59 token longest repetition runsWith the fix:1–3 token longest repetition runs

The generated token IDs also matched the known-good hardcoded-BOS control exactly.We then tested 28 real tokenizers and verified that existing models — Llama, Qwen, Mistral, Phi, Gemma, etc. — remained byte-identical before and after the change.

What I learned:

1. LLM bugs don't always look like model bugs.If generation becomes repetitive or nonsensical, inspect the input IDs before changing sampling parameters or blaming the checkpoint

2. Special tokens are part of the model interfaceBOS/EOS tokens, tokenizer configuration, and chat templates aren't just preprocessing. They directly affect the sequence the model sees.

3. Fixing a tokenizer can introduce another bugAdding BOS globally might fix raw completion but create duplicate BOS when a chat template already emits it.

The right question isn't:“Should I add BOS?”It's:“Does the final sequence already contain BOS?”

4. Test token IDs, not just decoded textDecoded output can make a tokenizer bug look like a model-quality problem. Comparing the actual token sequence gives a cleaner signal.

One special token can completely change the behavior of an otherwise healthy LLM.

https://lnkd.in/gHbyK5Zthttps://github.com/unslothai/unsloth/pull/10312

#2 Embedding model loaded, embeddings API returned 501unslothai/unsloth#9149

The embedding model loaded successfully. The embeddings API still returned HTTP 501.

This one took me into a pretty interesting edge case in Unsloth AI Studio.

The setup was simple:Load an embedding model as GGUF.Call:/v1/embeddingsAnd get:501 — This server does not support embeddings. Start it with --embedding

But the model was fine.

The problem was how Studio decided whether to start llama-server in embedding mode.It was looking at one piece of GGUF metadata: pooling_type

If the header said MEAN, CLS, or LAST, we started with --embedding.The problem?That field is optional.

Some embedding models simply don't include it.So we had a perfectly valid embedding model that looked like an ordinary chat model to the launcher.

The interesting part was deciding what to do when the metadata is missing.The fix wasn't:“If pooling_type is missing, assume it's an embedding model.”

That would be dangerous.Instead, the detection now falls back to other signals:→ GGUF architecture→ model-name hints→ dedicated embedding architectures such as nomic-bert and llama-embed

But there's an important rule:Fallbacks can fill in missing information. They shouldn't override explicit information.So if the GGUF explicitly says NONE or RANK, we respect it.

That distinction ended up being important during review too. A generic bert architecture can also represent reranker/classifier models, so the fallback had to be conservative and fail closed when the evidence was ambiguous.

The result:nomic-embed-text-v1.5Before: HTTP 501After: HTTP 200 + 768-dimensional embeddings

We also added tests for GGUFs without pooling_type, including synthetic embedding models, and ran an A/B comparison where the old implementation had 27 failures and the fixed version had 312/312 tests passing.

What I took away from this:

1. Optional metadata should actually be treated as optional.If your system depends on a field that isn't guaranteed to exist, you need a fallback strategy.

2. Fallbacks need precedence rules.Use secondary signals to fill gaps , never to silently override explicit metadata.

3. Model detection is a classification problem.Architecture, metadata, naming, and model behavior can all be signals. No single field should automatically become the entire truth.

4. Fail closed when the evidence is ambiguous.A false positive can be just as problematic as a false negative , especially when embedding and reranking models can share architectural characteristics.

This is one of those bugs where the model itself wasn't broken, Our assumptions about the model metadata were

https://lnkd.in/gZcZUbqChttps://lnkd.in/gQsQ_hD9https://github.com/unslothai/unsloth/pull/9149

#3 token=False still used my Hugging Face tokenunslothai/unsloth

“I passed token=False. It used my token anyway.”

I found this while working on the LoRA → GGUF export path in Unsloth AI.

At first, it looked like a simple argument-forwarding bug.save_pretrained_gguf(..., token=False) wasn't forwarding the argument to the LoRA save path.So the callee fell back to get_token() and picked up the operator's Hugging Face credential.

Easy fix.Except it wasn't.

The converter runs in a child process with os.environ.copy().So even after fixing the function argument, the child process could still inherit the operator's credentials.

That led to the bigger problem:Passing token=False is not enough if the process that actually performs the operation still has access to the credential.

There were also other credential paths to account for:→ token aliases→ HF_TOKEN_PATH→ OIDC environment variables→ another converter with the same environment handling→ an imatrix path that resolved the token too early

The fix was to make the credential boundary explicit:→ str → use this token→ None → ambient credentials are allowed→ False → credentials are explicitly denied

For the denied case, the subprocess environment is actively scrubbed instead of simply omitting a token.

Then came another important change.Instead of writing a test that only reproduces this particular bug, I added an AST test that checks for the dangerous pattern:os.environ.copy() → child process

If a converter creates its own environment without going through the credential boundary, the test fails.That caught another instance of the same problem.

What I think is useful is

1. API-level security controls are not enough.If your function eventually launches a subprocess, the subprocess is part of your security boundary.Check what it inherits.

2. “Not provided” and “explicitly denied” are different states.Using None, False, and a real credential as three distinct states prevented an important class of fallback bugs.

3. Test the pattern, not just the incident.A regression test can prove that this bug is fixed.A structural test can help prevent the next version of the same bug.

4. Audit indirect credential paths.Environment variables aren't the only source.Token files, aliases, cached credentials, and identity/OIDC variables can all change what a subprocess is actually capable of doing.

5. Be precise about the security boundary.This fix prevents the child from sending the operator's credential to the Hub.It does not mean the child has zero access to every piece of locally cached Hugging Face data. A private config already present in the shared cache remains readable.

We verified token=False across huggingface_hub 0.34 → 1.30:216/216 in-process cases90/90 wire-level cases

No authorization header was emitted in the denied case.

The thing I took away from this:A security boundary isn't where you say “no.”It's where the credential can actually cross the boundary

#4 SSH sandbox bypass via Python librariesunslothai/unsloth#10642

The agent sandbox blocked the ssh command. It did not block ssh.

I found this while working on the AI agent tool-calling sandbox in Unsloth Studio's backend.

The rule was simple:ssh, scp and sftp were hard-blocked at execution time.Even after the user approved the tool call.

So the policy looked airtight.

Then I looked at the Python side.paramiko, asyncssh and fabric could open the same connection.Especially when the hostname was a variable that static analysis couldn't resolve.

Same capability. Two paths. Only one of them had a guard.

Easy fix. Block the libraries too.Except it wasn't.

That would make the sandbox consistent, but also useless for the one legitimate workflow people actually want: an agent deploying to a server the user has reviewed and approved.

A blanket ban only pushes people toward bypasses.

So I replaced the hard blocklist with a per-session approved-server allowlist, and put both paths behind the same policy:→ Terminal: ssh, scp, sftp→ Python: paramiko, asyncssh, fabric connect patterns→ Dynamic hosts (ssh $HOST, connect(hostname)) fail closed until a literal host is approved→ Allowing a tool call records that target host for the session only

The model proposes a command. The user sees it and approves it. The host is recorded. Later calls to that host work. Everything else stays blocked.

I also exposed small API endpoints to approve and list hosts per session, and added a large regression suite covering host extraction from shell commands and Python AST, gating, and execution.

What I learned:

1. A security policy has to cover the capability, not the tool name.If you block a CLI but not the library that does the same thing, you've blocked a spelling.

2. Dynamic values are where guards quietly fail.If you can't resolve a host statically, the safe answer is "not approved yet", not "probably fine".

3. Hard blocks and approvals are different tools.A hard block says never. An allowlist says only what a human has seen. Agents need the second one.

4. Scope approvals narrowly.Per session, per host. Not forever, not for everything.

5. Put every path behind one policy.Two code paths with two implementations of the same rule will drift.

Agent safety isn't about saying no louder. It's about making "yes" specific.

#Unsloth #AIEngineering #AIAgents #Python #APISecurity

https://github.com/unslothai/unsloth/pull/10642

#5 Token-shaped string took the operator cache fast pathunslothai/unsloth#10264

I sent a fake API key. Unsloth Studio served me cached data anyway.

Working on Hugging Face Hub token authorization in the Unsloth Studio backend, I ran into this.

The earlier fix keyed every disk-cache gate on one check: is_anonymous(hf_token).

That is only true when the caller sent no token. So the gate was really asking "Did the caller send nothing?" and not "Is this credential allowed to see this repo?"

Any token-shaped string left the anonymous sentinel and took the same fast paths as the operator:→ cached dataset previews→ the chat template walk→ config and audio probes→ GGUF listings

A dummy string was enough.

There were many gates across routes, services and utils. Eight separate checks would drift apart, and the next endpoint would forget one. So I wrote one helper and applied it at all of them.

It treats the three caller states differently:→ the sentinel: no cache→ ambient or UI session: cache, with no extra Hub call→ explicit token: cache only after HfApi.repo_info succeeds for that specific repo

The answer is memoized for 60 seconds, so the Hub round trip isn't paid on every request.

Details that mattered:→ Unexpected types fail closed instead of falling into the ambient branch→ Local paths are never probed, so a caller's path and bearer token don't go over the wire→ Offline, an explicit token is denied rather than stalling on a probe

UI sessions keep the fast path. A made-up API key doesn't.

I added regression tests for the caller-identity cases: 84 passed. Two others need the datasets package and were skipped, same as before.

Four lessons:

1. A sentinel identifies a class of caller, not a credential."Not anonymous" is not the same as "authorized".

2. Gate on the resource, not the string.

3. Centralize the check. One helper at many gates beats many slightly different checks.

4. Default to deny.Anything unexpected should land on the closed side.

A cache is an authorization boundary too.

#Unsloth #BackendEngineering #APISecurity #HuggingFace #Python

https://github.com/unslothai/unsloth/pull/10264

#6 Batch retrieve leaked provider file idsBerriAI/litellm#34092

Every route in the LiteLLM batch flow worked. Then the results download returned a 500.

And that wasn't the worst part.

I traced it while working on the LiteLLM proxy, the LLM API gateway.

The flow:→ upload a JSONL, get a managed file id back→ create a batch→ poll until it completes

Then the completed batch response handed back a raw provider output_file_id. An OpenAI-style file-abc123, not the managed id the gateway had given out everywhere else.

Downloading it through the gateway failed, because no managed-file row existed for it. Worse, raw ids skip managed-file ownership checks.

On the old code, a different key on the proxy could download another user's batch output. I reproduced that against a live proxy with a real batch: the intruder got a 200.

A unified-id lookup on retrieve only works if the row already exists, and terminal batches persisted before output registration existed had none. So retrieve now registers the missing row itself.

Then the real question: who owns it? The caller who happened to retrieve it would hand the file to whoever polled first. Instead the row goes to the durable batch owner from the stored batch record. If there is no real owner on the record, registration is skipped rather than guessed.

Output and error ids now always come back as unified managed ids.

After the fix, on the same batch and database state:→ owner retrieve returns a unified id→ owner download returns 200→ intruder download returns 403

I added regression tests, including a cross-user denial test that runs the real managed-files hook.

What I took from it:

1. A convenience field can be an authorization bypass.A raw upstream id skips every check keyed on your own id.

2. Ownership belongs to the creator.Not whoever touched the resource last.

3. Heal missing state at the read path.Legacy rows exist in production.

4. Prove it with two identities.A single-user test can pass while another user can still read the data.

Still open: concurrent retrieves can mint separate ids for the same file. Both carry the same owner, so authorization holds.

An id the gateway didn't issue is an id the gateway can't protect.

#LiteLLM #BackendEngineering #APISecurity #LLM #Python

https://github.com/BerriAI/litellm/pull/34092

#7 Desktop SPA on loopback without opening the doorunslothai/unsloth#10847

http://127.0.0.1:8888 returned 404. The same app on the LAN IP worked.

I ran into this in the backend of the Unsloth Studio desktop app.

Desktop runs the backend with --api-only on 127.0.0.1. The packaged web UI was only served to the Cloudflare edge or the LAN listener.

So the one address you'd naturally try first, localhost, got nothing.

There was a second cost. Loopback is a browser secure context, so mic dictation through getUserMedia was blocked there too.

Quick fix. Serve the UI on loopback.The obvious fix had a catch.

"Looks like loopback" is exactly the kind of check that gets bypassed:→ a reverse proxy on the same machine makes remote traffic look local→ a forwarded header can claim anything→ a malicious page can rebind a hostname to 127.0.0.1

So the gate admits a request only if all of these hold:→ the listening socket is loopback→ the peer address is loopback→ the Host header is loopback→ no Forwarded, X-Forwarded-*, X-Real-IP or CF-Connecting-IP header is present

I reused the same check that already gates bootstrap password injection, rather than inventing a second definition of "local". Bootstrap injection itself is unchanged.

I ran a real uvicorn server and compared base against the fix:GET / via 127.0.0.1: 404 -> 200GET / with Host: localhost: 404 -> 200GET /chat deep link: 404 -> 200GET / with X-Forwarded-For: 404 -> 404GET / with CF-Connecting-IP: 404 -> 404GET / with Host: evil.example: 404 -> 404

The targeted test files ran 246 passed. Three updated tests fail on main.

What I learned:

1. Loopback is a combination of facts, not one.Socket, peer and Host all have to agree.

2. Proxy headers are evidence of a proxy.If they're present, it isn't a direct local browser.

3. The Host header is attacker-influenced.Checking it is what stops DNS rebinding.

4. Reuse the existing trust predicate.Two definitions of "local" will eventually disagree.

5. Test the rejections as carefully as the success.The 404s that stayed 404 were half the point.

Making something reachable is easy. Making it reachable only to the right caller is the work.

#Unsloth #BackendEngineering #APISecurity #FastAPI #Python

https://github.com/unslothai/unsloth/pull/10847

#8 Tool file worked in test-run, failed when publishedlanggenius/dify#41300

The workflow worked in the test-run. The published version sent the LLM no image.

Tracing an LLM workflow in Dify's backend, I hit this: a workflow that downloads an image in a tool node and passes it to a later LLM vision node.

In the orchestration test-run: fine.Invoked from an Agent as a published workflow: the prompt arrived with files: [].

No error. Just a missing image.

The difference was the file-access scope.→ Orchestration binds an Account scope, so plugin ToolFile rows are readable→ Agent chat binds an EndUser scope

And plugin uploads often store ToolFile.user_id as a different identity than the chatting end user. The later lookup then fails the ownership filter and the file silently drops out.

The obvious fix. Loosen the ownership check for tool files.It wasn't that simple.

That ownership filter is there for a reason. Relaxing it would let one user's file be read by another, and I'd have traded a missing image for a data-access problem.

So I followed a pattern Dify already has for upload files: an explicit grant.

When the tool message transformer produces a tool_file_id, it records that id on the current FileAccessScope. Later nodes in the same execution can attach those files.

The grant is:→ per execution, not global→ limited to files this run produced→ additive: ungranted tool files still require a matching user_id

I added tests for both sides: the transformer grants the file to the current end-user scope, and build_from_mapping accepts a granted tool file for a different end user. I also ran the existing file-access unit tests.

What I learned:

1. Silent failures are the expensive ones.An empty file list looks like success until someone reads the output.

2. Test-run and production can run under different identities.If a workflow passes in one scope, check it in the other.

3. Don't loosen a check to fix a symptom.Ask what the code legitimately needs access to, and grant exactly that.

4. Grant by provenance.A file produced by this run is a different thing from a file that merely exists.

5. Reuse the established pattern.Mirroring the upload-file grant kept the change small and easy to review.

The safest fix to an access bug is a narrower grant, not a weaker check.

#Dify #AIEngineering #LLM #AIAgents #Python

https://github.com/langgenius/dify/pull/41300

#9 Zotero RAG connector hardeninginfiniflow/ragflow#19742

Adding a new RAG data source looks like an integration task.With Zotero, it turned out to be a security task.

I added Zotero as a data source in the RAGFlow RAG pipeline, with connectors in both Python and Go.The happy path is simple:Pull the PDFs from a user's library.Parse them.Index them for retrieval.

Easy feature. The details mattered.

The connector takes a user-supplied WebDAV URL and an API key, and then makes outbound requests from the server.That is a textbook SSRF surface, plus a credential that must never end up at the wrong host.

The naive version:Fetch whatever URL the user typed.Send the API key with every request.Follow redirects.

Here is what I did instead:→ WebDAV is HTTPS-only, validated locally before any network call→ An SSRF check on the resolved address, with DNS pinned so the host can't change between the check and the request→ The Zotero API key is sent only to api.zotero.org, and redirects are followed manually without forwarding the key→ A cap on file redirects, and oversized attachments are skipped→ No default WebDAV URL, so we never claim Zotero hosts WebDAV somewhere it doesn't

The other problem was scale.A research library can be big, and buffering all of it in memory before indexing is a production reliability problem waiting to happen.So sync is incremental and batched: PDFs are yielded in the Python connector and downloaded per batch via NextBatch in the Go one.

What I learned:

1. A data connector is an outbound-request feature.Anything that fetches a user-controlled URL needs SSRF protection from day one.

2. Checking a hostname is not enough.If you resolve it twice, the answer can change. Validate the IP and pin it.

3. Credentials should be scoped to a host, not to a request.Redirects are where API keys quietly leak.

4. Validate configuration before touching the network.Fail fast on bad settings with a clear error.

5. Stream your sync.Batching keeps memory flat no matter how large the source is.

Good AI engineering isn't only about retrieval quality. Backend engineering means making sure the pipeline feeding your RAG system can't be turned against you.

An integration isn't done when it syncs. It's done when it fails safely.

#RAGFlow #AIEngineering #RAG #APISecurity #Python

https://github.com/infiniflow/ragflow/pull/19742

#10 Chat-template markup injectionunslothai/unsloth#7334

A string in a tool result could forge an assistant turn.No exploit. Just text.

The report that started this, in Unsloth's LLM chat-template handling, was small:A literal </think> in a user message was read as the end of the reasoning block.The rest of the model's thought leaked into the visible answer.

Then I looked closer.The same hole meant a tool result containing<|start|>assistant<|channel|>final<|message|>could open a whole assistant turn that the model never wrote.

Both were the same bug:Text the client controls reaches a chat template as markup instead of as content.

The fix looked simple. It wasn't.

The naive fix is to strip </think>.But the problem isn't one token. Every model family has its own control syntax:→ ChatML and Harmony style pipe tokens→ DeepSeek's fullwidth-bar tokens→ Mistral and Llama-2 [INST] brackets→ Qwen tool XML, including the tools catalog block→ Gemma turn delimiters

So the fix in Unsloth Studio neutralizes control markup with one small substitution, a space after the opener, so the text stays readable but is no longer a delimiter.

The harder question was where to apply it.Not just the message content. Also:→ tool result names→ replayed tool call arguments→ tool catalogs copied from remote MCP servers→ the render paths that bypass the shared template function, including llama-server passthrough and token counting

There were also trade-offs I wrote down instead of hiding.The marker list is closed on purpose, so ordinary text like <div> or [TODO] is untouched.Replayed assistant content only gets its turn boundaries broken, because rewriting its own think and tool markup would corrupt the transcript.And a tool whose name contains control markup is dropped with a warning, since rewriting a dispatch name would silently break the call.

For verification, I rendered 12 real upstream templates. A hostile paste forged structure in every one before the fix and in none after, while benign prompts rendered byte-identically.

What I learned:

1. Prompt injection isn't only semantic.It can be structural. The template is a parser, and the parser trusts its input.

2. Sanitize at every point where text reaches the template.One missed render path is the whole vulnerability.

3. Don't rewrite identities.Fields used for dispatch can't take the same treatment as content.

4. Test against the real templates, not a model of them.

5. Write the trade-offs down.A closed list with known costs beats a clever regex nobody can reason about.

If your template can't tell content from control, your user can.

#Unsloth #AIEngineering #PromptInjection #LLM #AISecurity

https://github.com/unslothai/unsloth/pull/7334

#11 Canonicalizing web_search argumentsunslothai/unsloth#9716

The model called web_search. Nothing was searched.

In Unsloth's LLM tool calling, small local models fail in quiet ways:{"q": "unsloth"} returned "No query provided." without searching.{"href": "https://example.com"} never fetched the page.{"query": 123} crashed with an 'int' object has no attribute 'strip'.

Other models say q, search_query, href, uri or link. The tool only understood query and url.

Worse, in the Approve for me mode a URL fetch is supposed to ask for approval. The check only read url, so the same call with a different key skipped the prompt.

My first idea was to tell the model the accepted argument names in the tool description.I ran an A/B across 5 models, and it wasn't a safe trade:it helped some, and hurt others. Llama-3.2-3B dropped from 88/105 correct calls to 75/105, with URL fetches falling from 30/30 to 19/30.So I reverted it. The schema the model sees is unchanged.

Here is what actually worked, on the server side:→ Heal known aliases to canonical query and url→ Ignore non-string values instead of passing them down→ Do it once, in prepare_call, before anything reads the arguments

Now the status text, the approval gate, the duplicate key and all three tool loops see the same arguments. An href call is gated and fetched.

The new tests fail on main and pass on the branch.

Takeaways:The model is the unreliable input, so normalize it at the edge.Approval and dedup must read the canonical form, or they disagree.Prompt changes are not free, so measure them across models.

How do you handle tool-call argument drift in your agents: heal it server-side, prompt around it, or reject and retry?

#Unsloth #AIEngineering #ToolCalling #AIAgents #LocalLLM

https://github.com/unslothai/unsloth/pull/9716

#12 Forced tool_choice on local GGUFunslothai/unsloth#9773

I forced the model to call web_search. It answered from memory instead.

The request to Unsloth's LLM tool-calling path was explicit:tools enabled, web_search selected, tool_choice forced.On a local GGUF model, no tool ran.Leave the selection out and web_search worked fine.

My first suspicion was the hosted-tool logic that decides which tools stay local.It was the wrong place.A loaded GGUF has no provider type, and the local catalog already kept web_search.

The real cause was one line.The agentic loop hardcoded tool_choice to "auto".So the forced choice never reached llama-server. The client asked for one thing, and the backend quietly sent another.

I expected a quick patch. The details mattered.

Just forwarding the value would introduce a new bug.If you force a tool on every turn, the model can never stop calling it and answer in prose.

So the design is:→ Forward tool_choice through both the OpenAI chat and Anthropic routes→ Honour a forced function on the first turn→ After a tool actually runs, drop back to "auto" so the model can answer→ Validate that the forced tool is actually enabled→ Honour "none" as a real value

Hosted providers keep their behavior: web_search without run_tools_locally stays hosted.

I added tests for the forced-then-auto sequence at the llama-server payload level, for the issue-shaped request entering the GGUF loop with the forced choice, and for the hosted-versus-local discriminator.

What I learned:

1. A hardcoded default is a silent override.If a user-facing parameter exists, trace it all the way to the wire.

2. Don't trust the first suspect.I looked in the routing logic. The bug was in the loop.

3. Forcing is per-turn, not per-request.A forced tool call is an opening move, not a standing order.

4. Test the outgoing payload.The assertion that matters is what llama-server actually received.

5. Keep API parity.OpenAI-style and Anthropic-style routes should behave the same for the same intent.

An option that's accepted but never forwarded is worse than one that's rejected.

#Unsloth #AIEngineering #ToolCalling #LLM #llamacpp

https://github.com/unslothai/unsloth/pull/9773

#13 max_completion_tokens retry for custom gatewaysunslothai/unsloth#10837

Every request failed with the same error:Unsupported parameter: 'max_tokens' is not supported with this model. Use 'max_completion_tokens' instead.

The setup: Unsloth Studio pointed at a custom OpenAI-compatible LLM API gateway in front of Azure-hosted gpt-5.x and o-series models.Unsloth Studio sent max_tokens.The gateway wanted max_completion_tokens.

The reason was simple.Only the real openai provider type sent max_completion_tokens.Everything else got max_tokens.

The obvious patch had a catch.

The naive fix: send max_completion_tokens for custom providers too.That breaks the other side.vLLM, llama.cpp and many other servers only know max_tokens."OpenAI-compatible" is a spectrum, and I can't tell where a gateway sits on it from its URL.

So I made the client learn from the server instead:→ The first request is unchanged, so servers that only know max_tokens see no difference→ If the upstream answers 400 and the error actually names max_tokens, resend once with max_completion_tokens→ Match on the structured error first (param and code), with a substring fallback for non-JSON bodies→ Any other 400 surfaces exactly as before, and a failing retry does too

Streaming was the tricky part.The retry has to happen before any bytes reach the client.The status check runs before anything is yielded, so the retry is invisible, and the stream body itself is untouched.

For tests, I drove the real client against a loopback HTTP server: stream and non-stream retry on the wire, an unrelated 400 is not retried, plus the error-matching cases. The two wire tests fail with the call sites reverted.

What I learned:

1. Compatibility is negotiated, not assumed.When you can't know the dialect up front, let the server tell you.

2. Keep the common path byte-identical.Retry logic should only exist on the failure path.

3. Retry narrowly.One retry, only on a specific error, only for a specific parameter.

4. Streaming changes where you can recover.Retry before the first yield, or you can't hide it.

5. Test over real HTTP.Mocking the client would never have proven the wire behavior.

The best compatibility layer is one the user never notices.

#Unsloth #AIEngineering #LLM #OpenAI #AzureOpenAI

https://github.com/unslothai/unsloth/pull/10837

#14 Wiki pages never retrievedinfiniflow/ragflow#18787

The wiki pages were in the index. They showed up in the UI. They never came back as RAG context.

The gap was in the RAG pipeline of RAGFlow, in its Knowledge Compilation feature.

Compiled wiki pages were browsable in Artifacts.But when I queried the knowledge base, retrieval acted like they didn't exist.

Nothing crashed.No error.Just silent misses.

The graph rows from the same pipeline (wiki_entity, wiki_relation) were retrievable.The pages weren't.

That difference was the clue.

The retriever always filters on available_int=1.The incremental wiki persist path wrote pages and canonical entities without available_int, without doc_id, and without content_with_weight.

So the data was there, but every retrieval gate excluded it.

Quick fix. Just stamp the fields.It wasn't that simple.

There was a second gate: _prune_deleted_chunks.It drops hits whose parent document no longer exists.Wiki pages use the KB id as a sentinel doc_id, or have none at all, so once source-document chunks were in the same result set, the pages got pruned as "orphans".

And the naive fix there is also wrong."Keep everything with a missing doc_id" would let stale chunks from deleted documents back in.

So the final design:→ persist stamps available_int=1, doc_id=kb_id and content_with_weight on pages and canonical entities→ FINALIZE backfills those fields on re-run, so existing pages become retrievable without a full rewrite→ pruning keeps KB-scoped rows (doc_id equals the KB id, or an explicit compilation marker)→ rows from a deleted source document are still dropped

I added regression tests for pages mixed with live and deleted source chunks, pages that omit doc_id, and the FINALIZE backfill.

What I learned:

1. "Visible" and "retrievable" are different properties.If the UI reads one path and the retriever reads another, a feature can look finished while doing nothing for the LLM.

2. Retrieval filters are part of your schema contract.Every field a filter touches has to be written by every producer, not just the first one you built.

3. Compare the rows that work with the rows that don't.Graph rows stamped the fields. Pages didn't. That diff pointed straight at the cause.

4. Cleanup logic needs an explicit scope."Missing parent means stale" is only true if you define which rows have a parent in the first place.

5. Ship the backfill with the fix.Fixing future writes does nothing for data that's already in the index.

A RAG system doesn't fail when retrieval returns the wrong thing. Sometimes it fails by quietly never returning the right one.

#RAGFlow #AIEngineering #RAG #LLM #Python

https://github.com/infiniflow/ragflow/pull/18787

#15 Credential guard hangs the terminal toolunslothai/unsloth#12087

The terminal tool call never returned. It just stayed on "Running" past the max duration.

This was in Unsloth Studio's LLM agent tool calling, and the cause wasn't the shell command.It was the safety check in front of it.

The setup: before an LLM-issued command runs, a guard scans it for references to Studio's credential files.To catch tricks like d=../..; cat "$d/auth.db", the guard expands shell assignments first.

Then someone runs something harmless:

echo "A=$A B=$B"

The assignment A=$A B=$B refers to itself.The expansion fed its output back into itself.It recursed without bound.

And it ran synchronously on the backend event loop, holding the GIL.So no tool result was logged, no completion reached the UI, and the whole server was stuck behind a log message.

Easy fix. Add a recursion limit.Except it wasn't.

A flat limit alone creates a different problem:→ too low, and legitimate long alias chains get refused→ too high, and you still burn CPU on the event loop→ and if hitting the limit means "allow", you've built a bypass

So the fix has three parts:→ resolve a self-reference against the earlier binding (or empty) instead of expanding it into itself→ bound the depth and the growth of the expanded text, while still allowing long finite chains→ when the budget runs out, fail closed and refuse the command

I also made the guard scan assignment-like text inside quotes separately, so a log message shaped like an assignment can't overwrite a real binding, while real quoted shell programs are still inspected.

Regression tests cover quoted self-references, long credential and harmless alias chains, scan exhaustion, and growing unresolved assignments.

What I learned:

1. Security checks are code on your hot path.A guard that can hang is an availability bug, and in an agent loop that means every tool call stalls.

2. Interpreters need budgets.Any time you expand, resolve or evaluate user-influenced input, put a hard bound on depth and size.

3. Fail closed when the budget is exhausted.If the guard can't finish its analysis, it can't claim the command is safe.

4. Don't let blocking work sit on the event loop.One synchronous scan can freeze every other request.

5. Self-reference is a normal input.A=$A isn't clever, it's just shell. Treat it as a case to resolve, not a case to hope doesn't happen.

A safety check that can hang is not a safety check. It's an outage with extra steps.

#Unsloth #AIEngineering #AIAgents #LLM #Python

https://github.com/unslothai/unsloth/pull/12087

#16 Three endpoints, three answersunslothai/unsloth#7375

The model was on disk. The model picker listed it. Auto-load said nothing was downloaded, then downloaded a different model instead.

A user report against Unsloth Studio's model loading backend got me started, with a Gemma-4 GGUF in the normal Hugging Face cache.

First I asked three endpoints the same question:

→ /api/hub/cached-gguf: 0 repos→ /api/models/local: model present, chattable→ /api/hub/gguf-variants: variant present, downloaded

That disagreement was the actual bug report.

My first theory, that the sweep couldn't see a custom folder, was wrong. Reproducing the user's layout disproved it, so I rewrote the PR around what I measured.

/api/hub/cached-gguf uses huggingface_hub's scan_cache_dir. The picker walks the directories directly. scan_cache_dir raises CorruptedCacheException for a repo whose refs/<branch> points at a commit with no snapshot directory, drops that repo, and only reports it in warnings. So the endpoint returned a clean 200 with an empty list.

Studio can create this state itself: a metadata probe that 404s can write refs/main at the live upstream HEAD without materialising that snapshot.

The fix came in two pieces:→ prune only refs whose commit has no snapshot on disk, then rescan once, and only when the scan warned→ stop the auto-load sweep from falling through to a Hub download after a load failed, and surface the model name and the backend's reason

Every catch had been parameterless, so the real error was discarded and the user got a success toast for a model they never asked for.

Tests cover the dangling ref, a healthy cache that must not be touched, an in-flight download, and a read-only refs directory.

1. When two views of the same data disagree, query both.

2. Tools that skip bad entries must say so loudly.

3. Throw away your first theory when the repro disagrees.

When three endpoints disagree, the bug usually lives in the one that fails quietly.

#Unsloth #AIEngineering #HuggingFace #LLM #Python

https://github.com/unslothai/unsloth/pull/7375

#17 Session not found after webhook disconnectlanggenius/dify-plugin-daemon#820

The workflow succeeded in Dify. The plugin still got "session not found".

This one came from Dify's plugin daemon, the Go plugin runtime behind it, and it only showed up with a certain kind of client.

Endpoint plugins, like a WeChat Work bot, receive webhooks with short callback timeouts.The client gives up and disconnects while the plugin is still doing a backwards invocation, such as session.app.chat.invoke.

The invocation was fine.The plumbing underneath it wasn't.

Two things happened on disconnect:→ the HTTP request context was cancelled, and a context.AfterFunc closed the plugin invocation stream, which removed the session from the session map→ the handler returned immediately on CloseNotify(), closing the response stream before the plugin finished

So the plugin tried to call back into Dify, and the session it was calling through was already gone.

The obvious fix. Ignore the disconnect.The details mattered.

If you simply ignore cancellation, you've created a session that can live forever.And the request context also carries trace metadata I didn't want to lose.

So the design is:→ build the session context with context.WithoutCancel, which keeps the trace values but detaches from the client's cancellation→ wrap it in context.WithTimeout using maxExecutionTime, so the session still has a hard upper bound→ on client disconnect, wait for the handler to finish or for maxExecutionTime before tearing down the stream

I added unit tests that cancel the HTTP context and check the session context survives, and that the session context still times out at its deadline.

What I learned:

1. The client's lifetime isn't the work's lifetime.If the work has side effects beyond the HTTP response, don't tie it to the connection.

2. Detach cancellation, not values.context.WithoutCancel keeps tracing and request metadata while removing the cancel signal.

3. Every detached context needs its own deadline.Removing one bound means adding another, or you've introduced a leak.

4. Cleanup hooks fire on the path you didn't plan for.An AfterFunc that tears down state is only correct if cancellation always means "we're done".

5. Short-timeout callers are normal.Messaging platforms often disconnect long before an LLM-backed workflow finishes. Design for it.

A disconnected client is a normal event. Your session shouldn't treat it as a shutdown.

#Dify #BackendEngineering #Golang #LLM #ProductionReliability

https://github.com/langgenius/dify-plugin-daemon/pull/820

#18 Upgrade deleted the working installlanggenius/dify-plugin-daemon#831

A plugin upgrade could remove the version that was working, and replace it with one that never started.

Looking at a serverless upgrade failure in the Go backend of Dify's plugin daemon, I found a shortcut. If the new version's package had already been uploaded, UpgradePlugin took a synchronous route:→ update the tenant installation records→ possibly delete the previous version→ return success

What it did not do was make sure the serverless runtime for the new version existed.

So if Lambda creation failed, say because the container image tag wasn't there yet, the database had already moved on. The user's working install could disappear from the UI while the new runner never came up.

The failure wasn't in the runtime launch. It was in the order of operations. ProcessUpgradeJob launches or verifies the runtime first, and only then calls curd.UpgradePlugin.

So I removed the shortcut. The upgrade always goes through the job, and when the package is already uploaded I still fetch its declaration and pass it in.

The result is a single rule: verify the new thing works before you mutate the state that points at the old thing.

To be precise, this doesn't fix why the image was missing in the first place. That's a separate build pipeline concern. What it fixes is the damage: a failed launch no longer takes a working install with it.

Three lessons:

1. Fast paths are where invariants go to die.2. Verify first, commit second.3. Delete a code path before you patch it.

An upgrade that can't roll back shouldn't commit before it knows it worked.

#Dify #BackendEngineering #Golang #Serverless #ProductionReliability

https://github.com/langgenius/dify-plugin-daemon/pull/831

#19 Live dataset preview mid-rununslothai/unsloth#10737

The data tab said "Generating data..." for the entire run.The rows were already on disk.

I hit this in the Unsloth Studio backend, while working on its data recipe runs for synthetic LLM data generation.A full run with batching writes parquet shards as each batch finishes.But the UI had nothing to show until the whole job was done.On a big run, that means staring at an empty state with no idea if your recipe is producing good data.Against main, the dataset endpoint returned 404 mid-run.On my branch, with a 2M-row run at batch size 1000, it returned 40,000 rows while the job was still going.

Simple fix. Just read the partial parquet files.That fix would have created a new problem.

The backend didn't know where the output was going until the run finished.And reading files that a worker is still writing is a different problem from reading finished files.

So the design had a few parts:→ The backend publishes the planned artifact path when the job starts, so polling knows where to look→ The frontend refreshes the data tab on a short interval while a run is active→ Pagination stays disabled during an in-progress full run and comes back after completion→ Pages are read with DuckDB, not a naive load of everything

Two details mattered most.

First, Windows.DuckDB opens files with FILE_SHARE_DELETE.The pyarrow fallback doesn't, and it would block the worker's merge step when it tries to remove the shard directory.A preview feature that makes the real job fail is worse than no preview.

Second, cost.Instead of scanning everything for each poll, I take row counts from the parquet footers, then scan only the shards the requested page actually covers.I also added tests that check paged reads match shards read in order.

What I took away:

1. A progress UI is part of the data pipeline.If users can't see partial output, they can't catch a bad recipe until it has already burned the whole run.

2. Readers must not interfere with writers.File sharing semantics on Windows can turn a harmless read into a failed merge.

3. Metadata is cheaper than data.Parquet footers already know the row counts. Use them.

4. Publish intent early.The planned artifact path at job start is what made polling possible at all.

5. Disable what you can't make correct yet.Pagination during an in-progress run stays off until it's stable.

A live preview is only a feature if it never gets in the way of the job it's previewing.

#Unsloth #BackendEngineering #LLM #Python #DuckDBhttps://github.com/unslothai/unsloth/pull/10737

#20 Prometheus metrics for plugin invocationslanggenius/dify-plugin-daemon#821

Our plugin daemon could tell you a plugin was called.It couldn't tell you how many were stuck right now.

I picked up an observability request for the Dify plugin daemon, the Go backend runtime for Dify's AI agent plugins.It already served Prometheus at /metrics when enabled.But if you were running agents and LLM tools in production, you still couldn't answer basic questions:→ How many invocations are in flight per plugin?→ How long do they take, and how did they end?→ What class of error is it?→ How many processes is each plugin running?

So I added four metrics around the main SSE invoke path:→ an in-flight gauge per plugin→ a duration histogram labelled by plugin and outcome (success, error, timeout, client_disconnect, session_error)→ an error counter labelled by error class (plugin_error, timeout, canceled, daemon_error, other)→ a per-plugin process gauge driven by runtime scale up and down

I also registered GET /metrics next to /metrics/ because many scrape configs expect the standard path.

Easy part: increment a counter.Hard part: make sure every exit path decrements the gauge, and that outcome labels mean something.

A client disconnect is not a plugin failure.A timeout is not a daemon bug.If you lump them into one error bucket, your dashboards lie.

Then there was a Go-specific trap.The scale callbacks receive a runtime through an interface.A nil pointer stored in an interface is not == nil.So a typed-nil runtime would slip past a plain nil check and panic when I tried to read its plugin ID.I added a guard that uses reflection to catch nil pointers, maps, slices and friends inside the interface before touching them.

I added tests for the in-flight lifecycle and for the mapping from outcome to error class.

What I learned:

1. Metrics are an API.Label names and values are a contract your alerts depend on. Choose them deliberately.

2. Separate outcome from error class.Duration needs to know how a call ended. Error budgets need to know why it failed.

3. Gauges leak if any path forgets to decrement.Test the lifecycle, not just the happy path.

4. Typed nil in Go interfaces is real.Observability code runs on every request. It must never be the thing that crashes the daemon.

5. Match conventions.Supporting the standard /metrics route saves everyone a config change.

You can't operate what you can't see, and you can't trust what you see if the labels are wrong.

#Dify #BackendEngineering #Golang #Prometheus #Observabilityhttps://github.com/langgenius/dify-plugin-daemon/pull/821

#21 Hidden reasoning prompt in vision fine-tunesunslothai/unsloth-zoo#1072

My vision fine-tune had a system prompt I never wrote."Reasoning effort is set to xhigh. Please think carefully..."

This came up while fine-tuning Qwen3.8 vision models with Unsloth Zoo, an LLM fine-tuning library.The chat template adds that system prompt by default.The only way to turn it off is to pass enable_thinking=False to apply_chat_template.

The text path already handled this.It merged per-row chat_template_kwargs from the dataset.

But UnslothVisionDataCollator called apply_chat_template with no kwargs at all.So for vision and audio fine-tunes there was no way to turn it off.Every training example carried that injected prompt, in both the input_ids and the labels.Nothing crashed. The loss went down. The model just trained on text I never intended.

Quick fix. Add a flag.Not quite.

The collator renders through two different layouts: a messages path and a prompt/completion path.If I patched only one, the other would keep silently training on the old behavior.

So the fix:→ a new optional chat_template_kwargs argument on the collator→ per-row kwargs from the dataset override the collator's, matching the text path→ both layouts render through one shared helper→ with no kwargs, the template call is identical to before, so default behavior is unchanged

For verification, I wrote tests that drive the collator on both layouts.The default sends no template kwargs.Collator and row enable_thinking=False reach every template call, and row values win.The 6 forwarding tests fail on main.I also ran the real Qwen3.8 chat template on a tiny Qwen2.5-VL processor and confirmed the reasoning prompt disappears from input_ids and labels.The existing vision collator suites still pass: 166 passed, 6 skipped.

What I learned:

1. Your training data is whatever the template renders.Not what your dataset says.

2. Parity between code paths is a feature.If text can do something, vision and audio should too.

3. Route duplicated logic through one helper.Two rendering paths means two places to forget a fix.

4. Defaults must stay boring.No kwargs should mean the exact same call as before.

5. Decode and inspect the final tokens.That is where a hidden prompt shows up.

The most dangerous prompt in your fine-tune is the one the template adds for you.

#Unsloth #AIEngineering #LLM #FineTuning #Pythonhttps://github.com/unslothai/unsloth-zoo/pull/1072

#22 Mistyped GGUF ids answered by the wrong modelunslothai/unsloth#8389

I sent an image to a vision model that didn't exist.Unsloth Studio's API server said the loaded model doesn't support vision.

That's the bug I fixed in Unsloth Studio's OpenAI-compatible /v1 API, the LLM serving backend.

A client called /v1/chat/completions with a mistyped vision model id.The server didn't say "I don't have that model".It quietly sent the request to whatever model happened to be loaded.That model was a text model, so the client got "Image provided but current GGUF model does not support vision".A real error message, pointing at the wrong problem.

The cause was in how the server decided to refuse a model name.It only rejected explicit quant references like :Q4_K_M, or ids it already knew locally.A bare hub-style id such as unsloth/typo-vision-GGUF that wasn't downloaded and wasn't in the catalog fell through to the drop-in compatibility path.

The obvious fix. Reject every unknown model.That would have been the wrong fix.

Plenty of clients send labels like openai/gpt-4o or anthropic/claude-3.5-sonnet and just expect the loaded model to answer.Breaking that would break the drop-in promise.

So I drew a narrower line.I added a helper that detects ids that clearly point at this server's GGUF catalog:→ repo names ending in -GGUF→ anything under the unsloth/ namespace

Those are treated as concrete references.If they don't match the loaded model, the API returns 404 with model_not_found.

Everything else keeps the old behavior:→ foreign provider labels still fall through to the loaded model→ an org/Unlisted id that isn't in the catalog still falls through

I added unit tests for the helper and the reject path, plus an integration test for the exact case: image plus mistyped id returns 404 before the vision guard.

What I took away:

1. Silent fallback is a design decision.If you route to a default, you are choosing to hide mistakes.

2. Wrong errors are worse than no errors.A misleading vision error sent the debugging in the wrong direction.

3. Compatibility needs a boundary.Be explicit about which ids you own and which you pass through.

4. Fail fast when the intent is clear.A name that targets your catalog is a request for that model.

5. Test the negative cases too.The passthrough cases deserve regression tests as much as the new 404.

An API that answers with the wrong model is lying. A 404 is honest.

#Unsloth #LLM #BackendEngineering #Python #APIDesignhttps://github.com/unslothai/unsloth/pull/8389

#23 Validate before you destroyunslothai/unsloth#7415

The request was rejected with a 400.But the model that was running was already gone.

This small ordering bug in Unsloth Studio's llama.cpp backend has a big blast radius.

If you load a DiffusionGemma GGUF with explicit gpu_ids on a Vulkan build, that combination isn't supported.The diffusion runner picks its device by CUDA physical index, and that has no defined mapping to Vulkan device ordinals.So rejecting it is correct.

The problem was when it was rejected.

load_model ran Phase 1 first, which calls _kill_process() and stops the active server.Only after that did it read the GGUF header and discover the model was a diffusion model.Then it returned the 400.

So a request that was always going to fail still tore down the healthy model.The user got an error and an empty server.

Simple fix. Move the check.The obvious fix had a catch.

You can't classify a GGUF without reading its header.And the header isn't always on disk.

So I handled each source:→ Local gguf_path: read the header with the existing diffusion check before killing the server→ Hugging Face repo: try a cached local GGUF first, and download only when it isn't cached, which was the existing behavior→ One shared error message and one helper, so the pre-teardown and later rejections can't drift apart

The old post-metadata check stays as a last line of defense.

I added tests that assert the local preflight sits before the teardown step, and that a rejection leaves the active server running.The existing remote diffusion preflight tests still pass.

What I learned:

1. Validate before you destroy.Every irreversible step should come after every check that can fail.

2. A failed request shouldn't have side effects.Rejection should be free.

3. Ordering bugs hide in happy-path code.Each step looked fine. The sequence was wrong.

4. Keep one source of truth for errors.Shared messages stop the same rule from being described three ways.

5. Keep the safety net.Early checks are for the common case, late checks are for the rest.

If a request is going to fail, it should fail before it breaks anything.

#Unsloth #BackendEngineering #llamacpp #LLM #Pythonhttps://github.com/unslothai/unsloth/pull/7415

#24 Resume button vanished after Stop and Saveunslothai/unsloth#8217

The training run was saved. The checkpoint was intact. The Resume button simply wasn't there.

This one is from Unsloth Studio's LLM fine-tuning workflow, while I was working on fine-tuning runs for Hub models.

The sequence was simple:Start a run.Hit Stop and Save early.Come back to resume.No button.

The checkpoint wasn't the problem.The problem was a status marker.

When a run starts, Studio writes an initial resource-provenance marker:status: pending

The worker later attests which model and dataset revisions it actually loaded.But if you stop before that happens, the marker stays pending.

And the resume check read "pending" as "model revision not attested."So it blocked resume, both in the UI and at resume start.

Not a one-line fix, it turned out.

The naive fix was: treat pending as fine and let everything resume.

That would quietly weaken a real safety check. Resume exists to avoid silently continuing a run against resources that changed underneath it.

So the question became:Is this run pending because it hasn't been attested yet, or because something is wrong?

The fix adds a small predicate, _provenance_awaiting_attestation(), that only matches the initial state:→ status is pending→ no model or dataset attestation status yet→ no snapshot paths recorded

Those runs resume like legacy runs without exact resource pins.

Everything else stays blocked:→ unattested snapshot paths→ incomplete provenance→ malformed markers

I added regression tests for the Stop-and-Save case, for the resume-start path, and for the case where untrusted snapshot paths must still be rejected. The resume and provenance suites passed.

What I learned:

1. A default state is not an error state."Not yet written" and "failed to verify" can look identical in a status field. They mean very different things.

2. Safety checks need to know which state they are looking at.Loosening a guard everywhere is the lazy fix. Narrowing the exact condition is the correct one.

3. Test the boundary, not just the bug.The regression test proves the button comes back. The negative test proves the guard still works.

4. User-visible bugs often live in backend state machines.The UI was correct. It was faithfully showing a wrong answer from the backend.

A guard that blocks the happy path by accident isn't safety. It's a bug wearing a safety badge.

#Unsloth #AIEngineering #FineTuning #LLM #Python

https://github.com/unslothai/unsloth/pull/8217

#25 Deep Research asked for more tokens than the context could holdunslothai/unsloth#7985

The model didn't hit an output cap. The error message said it did.

This is from Unsloth Studio's Deep Research feature, an LLM research workflow running against a local model.

The symptom:The synthesis step failed with finish_reason: length.The worker reported that the report reached its output limit.

But there was no output limit being hit.

What was actually happening:Deep Research clamped max_tokens against fixed ceilings, 8k for normal calls and 16k for synthesis.Meanwhile the loaded model had a context of 12288 tokens.

So synthesis could ask for 16384 completion tokens on a 12288-token context.Before generating a single word, the request was already bigger than the whole window.

The server truncated at the context wall and returned finish_reason: length.And the worker blamed an output cap that never existed.

The funny part:The code already knew the context length. It used it to size evidence and gate auto-scrape.It just never used it for the completion budget.

A quick patch would have been the wrong fix.

Clamping to context is not enough. The prompt also takes space.

So the fix does two things:→ max_tokens is clamped to context length minus the estimated prompt tokens, using the same chars-per-token heuristic synthesis already used for evidence budgeting→ the stream now captures usage, so a length failure can be classified

Then the error message tells the truth:→ context exhausted: tell the user to raise Context Length or reduce evidence→ true output cap: say so

I added tests for the context clamping, the usage shape from a live probe, and the updated error message. The Deep Research hardening suite passes. I did not run a full end-to-end Deep Research on a large cache with a loaded local model, and I said so in the PR.

What I learned:

1. finish_reason: length has more than one cause.Output cap and context exhaustion look the same unless you also look at usage.

2. Fixed ceilings are guesses.A constant that works on a large model silently breaks on a smaller one.

3. Use the number you already have.The context length was sitting right there, one function away.

4. Wrong error messages are bugs too.Sending a user to fix the wrong setting costs more time than the crash.

Budget the output against the window you actually loaded, not the one you hoped for.

#Unsloth #AIEngineering #LLM #AIAgents #Python

https://github.com/unslothai/unsloth/pull/7985

#26 Export silently pip-installed packages into the user's environmentunslothai/unsloth#9554, unslothai/unsloth#9405

I exported a model. My Python environment changed without me asking.

That's the LLM export bug I worked on in Unsloth's model export and quantization path.

When you export a model with FP8 or FP4 compressed-tensors, Unsloth needs llm-compressor.If it wasn't installed, the export just installed a pinned version on first use.

Convenient.Also a little scary.

Because the caller never agreed to have pip run in their environment.Maybe it's a production box. Maybe it's a carefully pinned env. Maybe it's a shared interpreter.

The first fix that came to mind was wrong.

The obvious fix: stop auto-installing.

That breaks everyone who relies on the current behavior.

So I split it into two layers.

Layer 1, the library (#9405):→ new install_missing_dependencies argument, default still True so nothing changes for existing callers→ pass False and it refuses to install→ instead it raises with the exact manual command for the active interpreter→ the flag is threaded through the merged-save and push-to-hub entry points→ the existing environment variable that disables auto-install still wins, even if the flag is True

Layer 2, Studio (#9554):→ a backend probe endpoint that checks whether llm-compressor is missing→ a consent dialog before compressed export, showing the target folder and the interpreter→ on accept, the choice is passed through the export pipeline→ the Studio export always passes the flag explicitly, because the library default is to install

One detail I like from review:Consent to provision a separate runtime is not consent to install into the current interpreter.So approving one doesn't silently approve the other.

For verification, the library PR has tests for the opt-out path, the env-var opt-out and the entry-point defaults. I also ran a GPU end-to-end export on a tiny Llama LoRA with llm-compressor absent: the default path installed it, the opt-out path raised the install command, wrote nothing and installed nothing.

What I learned:

1. Installing software is a side effect the user should be able to see."Helpful" and "surprising" are close neighbors.

2. Add the opt-out before removing the default.Backward compatibility and consent can both be honored.

3. Errors should be actionable.Raising with the exact pip command beats a vague "module not found."

4. Consent is specific.Agreeing to one install does not cover another one.

Convenience that changes someone's environment without asking isn't a feature. It's a surprise.

#Unsloth #AIEngineering #Quantization #LLM #Python

https://github.com/unslothai/unsloth/pull/9405https://github.com/unslothai/unsloth/pull/9554

#27 "Local" Claude traffic going to a remote gatewayunslothai/unsloth#9871

I told the agent to use my local model. The traffic went to a remote gateway.

This was in Unsloth's `unsloth claude` command and the local-agent child process, part of its AI agent tooling.

The idea is simple:Launch Claude Code and point ANTHROPIC_BASE_URL at a local server.

The launcher already scrubbed two things from the inherited environment:→ ANTHROPIC_API_KEY→ CLAUDE_CODE_OAUTH_TOKEN

I assumed that was enough.

It wasn't.

Claude Code also has provider-routing switches for Azure AI Foundry, Bedrock, Vertex and similar backends.And those take precedence over ANTHROPIC_BASE_URL.

So if the parent shell was configured for Foundry, the child process inherited that.Then "local" traffic quietly went to the remote gateway anyway.

No error.No warning.Just the wrong destination.

The fix looked simple. It wasn't.

The first thought was to add one more variable to the list.

But there were several routing paths, not one:→ the Foundry flag, base URL and resource variables→ the Bedrock and Vertex flags→ the Anthropic-on-AWS and Google Cloud flags→ the Mantle flag→ an inherited Unix socket setting

I extended the unset tuple to cover them.

And there were two launch paths:→ the main `unsloth claude` launch→ the subagent MCP bridge that spawns the local-agent child

They share the same tuple, so one change covers both. I also added tests that set these variables in the parent and assert they are absent from the child environment.

I reproduced the leak with Foundry variables present first, then confirmed they were gone after the fix.

What I learned:

1. Inheriting the environment is inheriting configuration.Every variable your parent shell has is a decision you did not make.

2. Precedence rules hide bugs.Setting the right base URL means nothing if another switch outranks it.

3. Scrub by intent, not by incident.Ask what could redirect traffic, not just which secret leaked this time.

4. Test every launch path.Two code paths with one shared list is good design. It still needs a test on each side.

If the destination matters, don't let the environment choose it.

#Unsloth #AIEngineering #ClaudeCode #AIAgents #APISecurity

https://github.com/unslothai/unsloth/pull/9871

#28 git not found in the Windows sandboxunslothai/unsloth#7323

git worked in every terminal on my machine. Inside the Studio sandbox it said: not recognized as an internal or external command.

This one took me into Unsloth Studio's sandboxed terminal tool for tool-calling AI agents, on Windows.

The model could only run git if it was told to use a short 8.3 path like C:\PROGRA~1\Git\cmd\git.exe.

The cause:The sandbox rebuilds PATH from scratch.On Windows, that meant the Studio venv plus System32.

Git for Windows lives under Program Files.So it never made it onto PATH.

Linux didn't have this problem, because /usr/bin was already there.

It looked like a one-line fix. It wasn't.

The naive fix: copy the host PATH into the sandbox.

That defeats the point of the sandbox.Host PATH can include user-writable folders, and a sandbox that auto-approves bare commands can end up running a lookalike binary dropped next to them.

The same goes for the working directory. cmd and CreateProcess look in the current folder before PATH for bare names.

So the design became about trust, not convenience:→ keep the curated prefix first, so Studio python and pip still win→ find the git the host shell resolves, and scan past an untrusted shim if needed→ append its directory only if it sits under a Program Files root→ resolve those roots with the Windows known-folder API, not environment variables a caller can override→ fail closed if that API is unavailable→ canonicalize the path, so 8.3 aliases and junctions resolve to the real location→ refuse user-writable locations such as per-user shims or temp folders→ restrict PATHEXT to .EXE and .COM, adding only the extension of the resolved git launcher→ set NoDefaultCurrentDirectoryInExePath so a workdir git.exe or rg.exe can't shadow anything

I added tests for each rule: the append order, the refused user-writable git directory, an env override not moving the trust boundary, the canonical path, scanning past an untrusted shim, and the fail-closed case.

What I learned:

1. "Works on my machine" can be a security feature.The sandbox was doing its job. The missing git was the cost.

2. Fix the usability problem without widening the boundary.Add one trusted thing, not everything.

3. Trust comes from the OS, not the environment.Environment variables are input. Known-folder APIs are authority.

4. Search order is part of the attack surface.Where a bare command resolves matters as much as what it resolves to.

A sandbox isn't useful if it can't run git. It isn't safe if it can run anything.

#Unsloth #AIEngineering #AIAgents #Python #APISecurity

https://github.com/unslothai/unsloth/pull/7323

#29 One help line, two false alarmsunslothai/unsloth#7332

Studio warned me that my llama.cpp build "lacks MTP support."The build supported MTP just fine.In Unsloth Studio's llama.cpp LLM inference backend, this was a good reminder of how much trust we put in CLI output.Studio probes llama-server by running --help and looking for the --spec-type option.The probe only read the first physical line that contained --spec-type.Some builds put the actual enum values on the indented description line below it.So the probe never saw them.Result: a capable build got a scary startup warning.I expected a quick patch. The details mattered.Making the parser read more lines is the obvious change.The real question was what to conclude when the probe can't tell.Before: an empty or crashed --help was treated the same as "no MTP."Now the probe parses the full --spec-type help block, declaration plus indented continuation.It only says MTP is missing when the enum is clearly listed without mtp or draft-mtp.If --help fails or comes back empty, the result is marked inconclusive and the warning stays quiet.While digging, I found a second bug in the same area.A bare llama-server crash with --mmproj still retries text-only, which is correct.But if that retry also failed, the user was told the vision projector was incompatible.On some ROCm/driver setups, that's just a crash. The projector was never the problem.Now the old wording is kept only for confirmed projector-format errors.Anything else is reported as a crash with --mmproj.I added tests for multiline help output, empty and crashed probes, an outdated build that should still report no MTP, and the retry messaging. I also checked against a real llama-server build.What I took away:1. Parse the whole block, not the first line.Help text wraps. A format that worked yesterday can move a value to the next line.2. "I couldn't tell" is a third state.Absence of evidence shouldn't be reported as evidence of absence.3. Fail open for warnings, fail closed for claims.A false alarm teaches users to ignore the real ones.4. Error messages are part of the API.Blaming the wrong component sends people down the wrong debugging path.A capability probe should only say "unsupported" when it has actually seen proof.

#Unsloth #BackendEngineering #llamacpp #LLM #Python

https://github.com/unslothai/unsloth/pull/7332

#30 Zod v4 schema lost its adaptermastra-ai/mastra#21187

An invalid tool call from an LLM crashed the Mastra AI agent.Not with a validation error. With:Cannot read properties of undefined (reading 'input')

The cause was in CoreToolBuilder. When background tasks are enabled, it injects extra fields into a tool's input schema: _background, suspendedToolRunId, resumeData. For Zod v4 schemas it does that through safeExtendZodObject.

That helper returns a fresh ZodObject, and the fresh object drops the custom ~standard.jsonSchema adapter that createTool() and toStandardSchema had installed. So the schema looked fine until something tried to convert it to JSON Schema. Then it blew up on the bad input.

A try/catch around the conversion would hide the crash and keep the root cause. So I fixed it where the mutation happens: after the fields are injected, the extended schema is re-wrapped with toStandardSchema, matching what the non-Zod-v4 fallback path already did.

I added a regression test that builds a Zod v4 tool, enables background tasks, and sends a malformed _background value. It now gets back a "Tool input validation failed" error, and the tool's execute function is never called.

Why this matters for agents: an LLM can recover from a clear error message, but not from a crash. Malformed arguments show up in production, so the unhappy path is the real path.

Agent reliability often comes down to what happens when the model is wrong.

#Mastra #AIEngineering #AIAgents #TypeScript #LLM

https://github.com/mastra-ai/mastra/pull/21187

#31 The second tool call vanishedmastra-ai/mastra#19940

My agent ran two tools in a row.

In Mastra, the TypeScript AI agent runtime, the second tool executed but its result was never shown to the model.

Nothing crashed. The next prompt just quietly lost both the tool call and its result.

The setup in Mastra's AgentController: non-yolo mode, two sequential tool calls, both auto-approved by an effective allow policy.

On the second resume, the controller hydrated its MessageList from the previous suspended snapshot. Validation had looked at the newer one, which contained the second tool call ID. But resumeStream and resumeGenerate still passed the stale one into resumeContext, so updateToolInvocation() couldn't find the second toolCallId.

Re-fetching at hydration time would add a second lookup that can disagree with the first. The real problem was two sources.

So #validateSuspendedToolCallTarget() now returns the suspended snapshot matching the resumed toolCallId, and resume hydrates from that exact snapshot. I also stopped relying only on status === "suspended" as the freshness check, since a stale snapshot has that status too.

The regression test runs two sequential allow-policy tool calls and asserts that the second tool call and its result survive into the final model prompt.

Execution counts said everything was fine while the model worked with half the history. Asserting on what the model actually sees catches that.

How do you test agent state: execution counts, or the exact prompt that reaches the model?

#Mastra #AIAgents #TypeScript #AIEngineering #LLM

https://github.com/mastra-ai/mastra/pull/19940

#32 Delete succeeded, the model stayedlanggenius/dify#39708

I deleted a custom model in Dify.Dify's backend API said success.The model was still there.This showed up after upgrading from 1.14 to 1.15 and later.Listing worked. Delete did nothing. And nothing failed, which made it harder to spot.The cause was a mismatch between how the app reads data and how it queries it.Older custom model rows can still store legacy model_type values:→ text-generation→ embeddings→ rerankingThe current code uses canonical values:→ llm→ text-embedding→ rerankReads go through EnumText, which normalizes the legacy values on the way out.So the list looked correct.But the delete lookup compared the column against the canonical value in SQL.Equality against "llm" doesn't match a row that says "text-generation".No row found. Nothing deleted. Success returned.Not a one-line fix, it turned out.A data migration is the obvious answer, but it doesn't help installations that haven't run it yet.And the read-side normalization means the app can't tell the rows are old.So the fix is on the query side.I added a small helper, _model_type_db_values(), that uses ModelType.to_origin_model_type() to return both the canonical value and its legacy alias.The custom model, credential, and model-setting lookups used by the delete flows now filter with IN over both values.I added tests that check the alias mapping for LLM, embedding, and rerank, and that the custom model lookup uses the legacy-aware filter.What I took away:1. Normalizing on read doesn't normalize in SQL.If an ORM type converts values on the way out, your WHERE clauses still see the raw column.2. "Success" with no rows affected is a bug smell.Delete paths should notice when they found nothing to delete.3. Upgrades leave old data behind.Code has to work against the database you actually have, not the one you wish you had.4. Read paths and write paths need the same assumptions.If listing works but delete doesn't, check how each one sees the data.A silent no-op is the hardest kind of failure to notice.

#Dify #BackendEngineering #Python #SQLAlchemy #LLM

https://github.com/langgenius/dify/pull/39708

#33 The answer rendered inside the thinking blocklanggenius/dify#41764

My agent's final answer showed up inside a collapsed "thinking" block.The model had answered correctly. The UI just treated the answer as reasoning.This happened in Dify's AI agent chat with reasoning models like GLM, which wrap chain-of-thought in <think> tags.The trigger was a tool call.Sometimes the model starts a think block and generation is interrupted by a tool call before </think> is ever emitted.Then a later <think> block, or the actual answer, ends up nested inside a tag that never closed.Markdown then renders an unclosed <details data-think>, and the reply is displayed as thinking.A quick patch would have been the wrong fix.The first idea is to fix it in the frontend only.But the unclosed tag is already in the text stream, and in stored output.So I fixed it in two places.In the Workflow Agent node, I added a streaming ThinkStreamState.It tracks whether a think block is open as chunks arrive, and inserts the missing </think> when:→ a tool log or link interrupts the stream→ a new <think> starts after real content→ the stream endsIn the markdown preprocessor, the same repair runs on the text itself, so already stored or LLM-tagged output still renders as valid think blocks.One detail I had to get right: a <think> that opens immediately after another with no content in between is just skipped, not treated as nesting.Tracking whether any non-whitespace content has appeared since the open tag handles this.I added tests around the agent node for text and tool-log chunks.What I learned:1. Streaming needs state.You can't repair a tag by looking at one chunk. You have to know what's still open.2. Model output is untrusted input.Reasoning models don't always produce well-formed markup, especially around tool calls.3. Repair at the source and at the render layer.Fixing the stream helps new output. Fixing the renderer helps what's already stored.4. Interruptions are the edge case.Tool calls break generation in the middle of structure, so structure has to be closed explicitly.Valid output and well-formed output aren't the same thing.

#Dify #AIEngineering #AIAgents #LLM #ReasoningModels

https://github.com/langgenius/dify/pull/41764

#34 CSV cells silently rewritten by pandaslanggenius/dify#41922

My CSV had the product code 00123.My Dify RAG knowledge base indexed 123.0.

Nothing crashed. No warning. The retrieval just quietly had the wrong data.

The cause sits in Dify's knowledge import path. The CSV extractor reads the file with pandas before building the text that gets indexed, and pandas tries to be helpful by default:

→ 00123 becomes 123.0→ a literal "NA" becomes nan→ a literal "NULL" or an empty cell becomes nan

Quoting the cells doesn't help, because the corruption happens after the file is parsed correctly.

So I changed the extractor's defaults:→ dtype=str, every cell stays the text it was in the file→ keep_default_na=False, "NA" and "NULL" stay as strings

Any explicit csv_args passed by a caller still override them, so typed parsing is still available.

Before: code: 123.0;region: nanAfter: code: 00123;region: NA

I added unit tests for preserved cell text, source metadata handling, and the explicit-override path.

For analytics, type inference is a feature. For RAG ingestion, the source text is the data, so a library default built for one job can quietly break the other.

In RAG, the quality of your answers can be decided before the first embedding is computed.

#Dify #AIEngineering #RAG #Python #LLM

https://github.com/langgenius/dify/pull/41922

#35 Excel parser thought full sheets were emptyinfiniflow/ragflow#19237

A spreadsheet full of data. RAGFlow parsed zero rows from it.

Another one lost everything after a blank gap in the middle.

I picked up a batch of document parser bugs in RAGFlow's RAG pipeline, and the first two shared a root cause.

The Excel parser needed to know where the real data ends. A sheet's max_row is often inflated by formatting far below the data, so the parser tried to be smart:

→ check the first 100 rows, and if they're blank, call the sheet empty→ binary search for the last row that has data

Looks reasonable. Except it wasn't.

Binary search assumes contiguous data. A blank preamble before the header, or a gap between two blocks, breaks that. The search lands in the gap and concludes everything after it doesn't exist.

Tuning the numbers would just move the failure to a bigger sheet. What I changed instead: openpyxl only materializes cells that exist in the file, so the new code scans those and takes the highest row with a non-blank value, ignoring styling-only cells.

Two smaller fixes went in the same PR:

→ UTF-8 BOM stripped before JSON/JSONL parsing, so Windows-exported files no longer silently return zero records→ the Go QA chunker routes XLSX JSON items whose text is a rendered HTML table through the table extractor instead of delimiter-based text parsing

Regression tests cover data below row 100, a blank gap with an inflated used range, BOM vs no-BOM, and the Go HTML-table path.

1. Heuristics that assume contiguous data fail on real files.

2. Ask the data structure, don't search it.

3. Silent zero-row output looks like an empty file, not a bug.

Document parsing is where a lot of RAG quality is quietly won or lost.

#RAGFlow #AIEngineering #RAG #Python #LLM

https://github.com/infiniflow/ragflow/pull/19237

#36 Autosave wiped a working pipelineinfiniflow/ragflow#18783

An editor tab left open. A backend restart. A Data Pipeline that was working yesterday is now empty.

Nobody clicked anything.

This was the bug behind a wiped pipeline report in RAGFlow's RAG pipeline editor, and it turned out to be the canvas autosave.

The autosave watched the graph store for changes to nodes and edges, and 20 seconds after any change it called saveGraph().

Including a change to [].

And an empty graph is exactly what the store looks like:→ before the agent has loaded→ after a remount→ when the backend returns a snapshot without a DSL

So the sequence was:Store resets to empty. Autosave sees a change. 20 seconds later it PUTs components: {} over a perfectly good pipeline.

Easy fix. Skip saving when the canvas is empty.Except it wasn't that simple, because an empty canvas can also be a legitimate state. A user may really have deleted everything.

So I separated the two cases.

Autosave now runs only when:→ the agent id is present→ the agent detail has loaded→ the chat drawer is closed→ the canvas has at least one node or edge

Explicit Save still persists an empty canvas.Automatic saves are guesses. User-initiated saves are intent.

Two more guards went in:→ the save hook no longer PUTs without an id→ the fetch hook doesn't apply the DSL to the store until a DSL actually exists, so a later empty response can't clear a loaded canvas

I pulled the decision into a small pure function, shouldAutosaveCanvas, and added tests for each skip condition, plus fetch-hook tests showing an empty response doesn't overwrite a loaded graph. One honest note: I could not run the full web Jest suite in my environment, which I called out in the PR.

What I took away:

1. Default state is not user state.An empty store means "not loaded yet" at least as often as "user cleared it".

2. Autosave is a write path, so it needs preconditions.Never write unless you know the state you're writing is real.

3. Separate automatic from explicit.The same action can be safe when a user asks for it and dangerous when a timer does it.

4. Put the decision in a pure function.It's easy to test every case when the rule isn't buried in a hook.

The scariest bugs are the ones that delete data while nobody is looking.

#RAGFlow #AIEngineering #TypeScript #React #RAG

https://github.com/infiniflow/ragflow/pull/18783

#37 Slow network blocked Explore pagelanggenius/dify#40176

The Explore page in Dify felt slow.On some networks it just hung, and the fault was in Dify's Python backend.

It wasn't the database or the LLM layer. It was a template fetch.

With HOSTED_FETCH_APP_TEMPLATES_MODE=remote, every Explore or Learn Dify request called tmpl.dify.ai with a bare httpx.get(). No cache. On a slow or blocked network, that blocked server-side rendering for up to the 10 second timeout.

I added an in-memory TTL cache around the remote template fetches. The design choices:

→ cache key is the URL plus the Origin header, which the existing code already forwards→ only successful 200 responses are cached→ TTL is configurable via HOSTED_FETCH_APP_TEMPLATES_CACHE_TTL, default 600 seconds, and 0 disables caching→ access goes through a lock, since Flask serves concurrent requests

Tests cover one network hit for repeated calls, failures not cached, TTL 0, and separate origins.

The takeaway: a remote call on the request path needs a budget, and a cache around it needs rules for what goes in the key, what never gets stored, and how to turn it off.

Reliability work in an AI platform isn't only about models. A lot of it is about the boring HTTP calls around them.

#Dify #BackendEngineering #Python #Caching #LLM

https://github.com/langgenius/dify/pull/40176

#38 local_files_only still hit the networkunslothai/unsloth#7482

I set local_files_only=True.The tokenizer files were already in the local cache.Unsloth, the LLM fine-tuning library, still tried to reach the Hugging Face Hub.

For anyone working offline, on an air-gapped box, or behind a strict firewall, that's a hard failure.

I traced it while working on Unsloth's offline tokenizer loading.

On transformers 4.57.2 through 5.5.4, PreTrainedTokenizerFast.from_pretrained(repo_id, local_files_only=True) can still call model_info() when no offline environment variable is set.That window includes 4.57.6, which Studio pins.

So the flag said "don't touch the network."The library touched the network anyway.

Upstream fixed this properly in transformers 5.6.0. But we can't just say "upgrade" while supported versions sit below it.

Easy fix. Set HF_HUB_OFFLINE for the user.Except that changes global process behavior for something that's only meant to affect one load.

So the workaround avoids the repo id entirely:→ resolve the cached snapshot with hf_hub_download(..., local_files_only=True)→ load the tokenizer from that local directory path→ a local path never triggers the Hub metadata probe

I wired it into the tokenizer loader, the vision processor fallback, and the save path, where a local cache check for tokenizer.model now happens before HfApi.model_info when offline.The helper's defaults stay closed, so it can't accidentally download with backoff when called without the flag.

I also added unit tests, wired into CI, plus integration tests that block the network and are gated on a populated local cache.

And I was explicit in the PR about scope: this does not close the reporter's full failure. That root cause lives in unsloth_zoo and stays tracked separately.

What I took away:

1. A flag documents intent, not behavior.Test offline mode with the network actually blocked.

2. Version windows matter.A bug can live across a range of dependency versions you still have to support.

3. Workarounds should be temporary and labeled.Say which upstream fix retires them.

4. Be honest about scope.A PR that fixes one layer shouldn't imply it fixes the whole bug.

"Offline" only means something if the network is really off the table.

#Unsloth #AIEngineering #HuggingFace #Python #LLM

https://github.com/unslothai/unsloth/pull/7482

#39 Streaming markdown backslash lineunslothai/unsloth#11501

One long line of backslashes froze my streaming chat UI for seconds.Not a crash. Not an error. Just a render in Unsloth Studio's local LLM chat that never seemed to finish.

This one came up in Unsloth Studio, the chat UI for running LLMs locally.Paste a Windows path, a regex, or escaped JSON into a message.Make it long.And every streamed render started costing seconds.

My first suspicion was the model, or the streaming transport.It was neither.

The cost was in how we split markdown into blocks.Streaming LLM output is re-rendered constantly, so the renderer splits the text into blocks and only re-renders the ones that changed.To split blocks, we called marked's Lexer.lex.That runs the block lexer AND the inline lexer.We only needed block boundaries.The inline pass was pure waste, and on a long line full of backslash escapes it was the expensive part.

Easy fix: skip the inline pass.Except it wasn't, because the splitter we were calling lived inside a dependency.

So the fix was:→ Vendor a block-only splitter that calls blockTokens instead of Lexer.lex→ Keep the same merge rules (HTML tag stacks, $$ math blocks, footnotes, CRLF) as the streamdown version it replaces→ Wire the streaming scheduler, the streaming markdown path and the LaTeX path to it→ Pin marked to 17.0.6 so it matches the version streamdown uses

The important part was proving I hadn't changed behavior.The test suite compares my splitter against streamdown's on a set of samples: headings, code fences, unclosed $$, nested HTML, tables, CRLF, footnotes and long backslash lines.A separate perf test feeds a 200,000-character backslash line and asserts the split finishes in under 500ms.

What I took away:

1. Know which layer of a parser you actually need.If you only want block boundaries, don't pay for inline parsing.

2. Streaming multiplies every constant.A cost you'd never notice once becomes a freeze when the UI re-renders on every token batch.

3. Adversarial-looking input is normal input.Paths, regexes and escaped JSON are everyday content in AI chat apps.

4. Vendoring needs an equivalence test.If you copy a function to change its behavior, prove it still matches the original everywhere you didn't mean to change it.

The fastest render is the one that doesn't do work you never needed.

#Unsloth #AIEngineering #TypeScript #LLM #Streaming

https://github.com/unslothai/unsloth/pull/11501

#40 Context bar after model loadunslothai/unsloth#7453

I loaded a new model. The context usage bar went blank.Nothing was broken. The bar just had no idea how full the chat was.

This was in Unsloth Studio, the local LLM app, and it looked like a small UI gap in context window token counting.The X / Y token bar was only ever written from streaming completion metadata.Loading a model cleared it.So until you sent another message, you couldn't tell whether your existing chat still fit the model you had just loaded.

Quick fix. Just recount the thread after the load.That would have been the wrong fix.

A background recount is a different animal from a completion:→ it has no abort signal→ it runs while the user keeps clicking→ it can finish after the world has moved on

So the design became mostly about what the recount must refuse to do.

On the backend I added POST /api/inference/chat/count_tokens. It prices OpenAI-form messages with the loaded GGUF tokenizer, through the same sanitization the completion path runs.It never auto-switches models. If it honoured a stale model name, a recount that started on model A could drag the backend back to A after you loaded B.Review pushed it further. It refuses while a generation is in flight, and the count stands down between its two llama-server calls if one starts mid-count.It also declines images and audio instead of returning a number that is too small.

On the frontend, refreshContextUsage() rebuilds the displayed branch from stored history and drops its own result if:→ the checkpoint moved→ the thread changed→ a real completion wrote usage while it was in flight

For verification, I added a negative control: reverting only the runtime provider change makes the new cases fail on a wrong value, not on a missing symbol.

What I took away:

1. A background job needs an invalidation story.Decide up front what makes its answer stale, then check that before publishing.

2. Read-only endpoints can still cause side effects.A "count" that triggers a model switch is not read-only.

3. An approximate number can be worse than no number.If the count would come back short, say nothing.

4. Prove your test can fail.A negative control tells you the test is guarding the behavior, not just the code shape.

This is the kind of LLM serving detail that never shows up in a demo, but decides whether a local inference UI can be trusted.

A number on screen is a claim. Don't publish it if you can't say it's still true.

#Unsloth #AIEngineering #LLM #TypeScript #Python

https://github.com/unslothai/unsloth/pull/7453

#41 Auto-compaction controlsunslothai/unsloth#9774

My chat history was being trimmed, and I couldn't turn it off.Local GGUF chats in Unsloth Studio, a local LLM inference app built on llama.cpp, always sent context_overflow: truncate_oldest.

So when the window filled, Studio compacted for you.By default it reset the conversation to a checkpoint.Or it dropped extra history, about 25% of the prompt budget, so the boundary wouldn't move on the very next turn.That's reasonable behavior. It's just not always the behavior you want.

The obvious fix. Add a toggle.The obvious fix had a catch.

The policy lived in rolling-context and checkpoint code, and the headroom value was in the middle of all of it.What I added to Chat settings:→ Auto-compact long chats, on by default→ When it's off, context_overflow isn't sent, so a full window errors instead of silently dropping history→ Reset conversation, or drop oldest turns→ Extra trim of roughly 25%, 10%, 5% or none

Under the hood these are per-request overrides: context_policy overrides UNSLOTH_CONTEXT_POLICY, and compaction_headroom_ratio overrides the process default.Requests that don't set them keep the existing defaults, so API clients are unchanged.

The part I cared about most was the headroom ratio.A UI slider is user input. If a bad value reached the backend, "extra trim" could eat the whole prompt.So the ratio is clamped server-side to [0, 0.9], and every per-request override goes through the same gate.The UI is a convenience. The server is the boundary.

I covered it with frontend tests for the settings and rolling window, and backend tests for the headroom override, for rolling policy winning over a checkpoint default, and for the settings payload bounds.

What I learned:

1. Silent data loss should be a choice.If a system drops context, users should be able to see it and turn it off.

2. Per-request overrides beat global flags.They give you control without breaking existing clients.

3. Validate at the server.A slider's range is a hint. The API has to enforce it anyway.

4. Defaults are part of the contract.Adding a setting shouldn't change what happens when it's unset.

LLM context management is product behavior, not just a token budget.

If the user can't see what was dropped, the context window is lying to them.

#Unsloth #AIEngineering #LLM #llamacpp #GGUF

https://github.com/unslothai/unsloth/pull/9774

#42 Vulkan backend for legacy AMD on Windowsunslothai/unsloth#7373

The installer worked fine. The GPU just wasn't there.In the Unsloth Studio installer for local LLM inference, an RX 480 on Windows AMD was invisible to llama-server.

The installer always picked the HIP prebuilt for AMD.Vulkan was only auto-selected for Intel and other GPUs.Cards below the HIP prebuilt's gfx floor, like gfx803, simply never showed up.

One user had a workaround.They copied the official Vulkan build over the bundled one, with an RX 9070 XT and an RX 480 in the same machine.It worked. Then the next Studio update put HIP back.

Easy fix. Prefer Vulkan on AMD.Except it wasn't.

That would break the mixed setup where the default card is HIP-supported. There, HIP is the right answer.So the design became two separate rules:

→ Explicit opt-in: UNSLOTH_LLAMA_BACKEND=vulkan (or --llama-backend vulkan) works on any host. The choice is stored as llama_backend in the prebuilt info file, so updates re-assert it.→ Auto-fallback: only when the active gfx arch, the device HIP can actually see, isn't supported by the HIP prebuilt.

Review kept making the fallback more careful:→ HIP_VISIBLE_DEVICES and similar masks change which card counts as active→ Family labels like gfx110X from update markers must keep HIP→ Fork-only gfx targets can't be treated as supported by other repos→ If no gfx arch is resolved, don't force Vulkan

Mixed setups stay on HIP by default. Vulkan there is opt-in.

For verification, I ran the routing and Vulkan update-preservation tests with mocked host profiles. I did not have a Windows AMD dual-GPU machine, so I said so in the PR and listed the manual checks I wanted before merge.

What I learned:

1. Fallback rules need a definition of "active".On multi-GPU hosts, the best device isn't always the one your runtime will use.

2. Explicit user choices should persist.If an update silently reverts it, it wasn't a setting.

3. Unknown should not mean unsupported.When detection fails, don't guess a backend.

4. Be honest about what you couldn't test.Hardware you don't own is a gap to name, not hide.

Good installers choose conservatively and remember what you asked for.

#Unsloth #AIEngineering #llamacpp #Vulkan #ROCm

https://github.com/unslothai/unsloth/pull/7373

#43 Rate-limit errors in llama.cpp updatesunslothai/unsloth#9980

The update failed. The error message was a wall of PATH variables.The actual reason was in the log. Just not in the part we showed.

Unsloth Studio's llama.cpp updater kept failing on GitHub API rate limits for some users.The updater pins a release tag, and that always went through api.github.com.Without GH_TOKEN, unauthenticated calls hit rate limits with a 403 or 429.The installer exited with code 2, and on Windows the in-app updater doesn't fall back to a source build.

Then the UI made it worse.stream_installer kept only the last 1500 characters of stdout.The real line, "prebuilt fallback reason ... set GH_TOKEN", had been pushed out by a long system report.

Simple fix. Show more output.It wasn't that simple.

More output is still noise. The better question was why a common update needed the API at all.So the fix has two parts:

→ Pinned release tags now use the existing download-host CDN path, with no api.github.com call in the common case→ When something does fail, actionable lines are preferred over the raw tail

Review made the error handling stricter:→ A rate-limit message needs both GitHub and rate-limit wording. huggingface.co can also return a 429 on the validation model download, and telling that user to set GH_TOKEN wouldn't help→ A retry that succeeds shouldn't beat the installer's final verdict→ A GitHub 429 is a rate limit, not just a 403

I checked it with targeted pytest cases for the CDN routing, error formatting and updater messages. I also reproduced the original failure with a mocked 403 and a long PATH tail before the fix.

What I took away:

1. Rate limits are a normal operating condition.Design the common path so it doesn't depend on a quota.

2. Don't truncate the reason.The last N characters is not the same as the most useful N characters.

3. Error attribution matters.Blaming the wrong service sends users to the wrong fix.

4. Retries are not verdicts.Report what finally happened.

Production reliability is often a message problem: users can only fix what you tell them.

The best error message names the cause and the next step.

#Unsloth #BackendEngineering #llamacpp #Python #ProductionReliability

https://github.com/unslothai/unsloth/pull/9980

#44 MinerU server_url in RAGFlow Go parserinfiniflow/ragflow#19445

The PDF parser was configured correctly. The inference address never arrived.In RAGFlow's Go backend for RAG document ingestion, on the MinerU path, the vlm and hybrid http-client backends lost their downstream server URL.

The provider UI collected mineru_server_url.The Go code never forwarded it to MinerU.So http-client backends had a backend name and nowhere to send the work.

There were two call sites:→ The ingestion path, dispatchMinerUPDF and mineruStreamParse→ The PDF parser path, the async task client in mineru_local.go

Quick fix. Add a field to the request.The details mattered.

The value can come from several places, so the real work was deciding precedence:→ parser setup→ the MinerU provider api_key JSON→ the MINERU_SERVER_URL environment variable

I wrote a resolver that handles all three, and both Go paths send the result as the server_url multipart field, mirroring the Python contract.

I also added validation:→ Backend names are checked locally against the MinerU list: pipeline, vlm-engine, hybrid-engine, vlm-http-client, hybrid-http-client→ vlm-http-client and hybrid-http-client require a server URL→ A missing URL fails fast with a clear error

That last point is a deliberate behavior change, so I wrote a migration note in the PR. Deployments that used http-client backends without a URL will now fail at parse time until it is set.

Tests cover forwarding the field, reading it from the provider JSON, the missing-URL error, and invalid backend rejection on both paths. I also called out what I did not verify: live testing of the leftover async task client against a real MinerU service.

What I learned:

1. A config field the UI collects but the backend drops is a silent bug.Nothing crashes. The feature just doesn't work.

2. Cross-language rewrites need contract tests.Matching the Python behavior is the spec.

3. Define precedence once.A single resolver beats scattered fallbacks.

4. Fail early on missing required config.A clear error at parse time is better than a confusing failure downstream.

Good RAG pipelines depend on boring plumbing: the document parser has to reach the model it was configured to use.

Configuration that never reaches the worker is just decoration.

#RAGFlow #BackendEngineering #RAG #Golang #MinerU

https://github.com/infiniflow/ragflow/pull/19445

#45 Studio systemd serviceunslothai/unsloth#9308

A crash shouldn't mean someone has to SSH in and restart your LLM server by hand.

That was the ask behind a feature request to run Unsloth Studio, a local LLM server, as a Linux systemd service:Start on boot. Come back after a crash.

Sounds like a systemd unit file.The first fix that came to mind was wrong.

A unit file is the small part. The hard part is what a default installer is allowed to do to someone's machine.

So I made a few design calls:

→ It is opt-in. Nothing changes unless UNSLOTH_INSTALL_SYSTEMD=1 is set at install time. No new prompt, no new output, and the existing "Start Unsloth Studio now?" flow is untouched.→ It runs as a user service, not a root one, written to ~/.config/systemd/user.→ It uses Restart=on-failure, so a crash recovers but a deliberate stop stays stopped.→ It binds to 127.0.0.1 by default, same as running unsloth studio by hand. Exposing it to the LAN means explicitly setting UNSLOTH_SYSTEMD_HOST=0.0.0.0.

Then the sneaky parts.

User services normally only run while you're logged in. For boot without a login session you need loginctl enable-linger once. The installer can't quietly decide that for you, so it prints the hint instead.

Uninstall was the one I cared about most.If a script deletes a unit file, how does it know the file is the one it created?So the installer stamps its unit with a marker, and uninstall only disables and removes units that carry it. A hand-written unit with the same name is left alone.

Order matters too. Teardown has to stop the unit before the kill sweep, otherwise Restart=on-failure helpfully brings the server back in the middle of removal.

For tests, I covered the generated unit contents, escaping, the systemctl calls, that leaving the flag off leaves the install unchanged, and that no helper is run from the caller's working directory. There is also a check that the installer asks no new interactive question.

What I learned:

1. Opt-in is a feature.Auto-start is useful, but an installer that changes system state by default is a surprise. Make the risky path a conscious choice.

2. Secure defaults beat convenient defaults.Localhost binding means the convenient option can't accidentally become an exposed server.

3. Uninstall needs ownership proof.Only remove what you can prove you created. A marker turns a guess into a check.

4. Lifecycle order is part of correctness.A restart policy and a cleanup script can fight each other if you don't sequence them.

5. Print the manual step you can't automate.Some things like linger need the user's consent. Say so clearly.

Reliable software isn't only about surviving a crash. It's also about cleaning up after itself.

#Unsloth #BackendEngineering #Linux #systemd #LLM

https://github.com/unslothai/unsloth/pull/9308

#46 Kokoro TTS gateway probe and playbackunslothai/unsloth#9905

The TTS server returned HTTP 200. The browser still refused to play it.

The error was: "The fetching process for the media resource was aborted by the user agent at the user's request."

I hit this in the backend text-to-speech integration, connecting Unsloth Studio to Kokoro FastAPI, an OpenAI-compatible TTS API gateway.

Two things were broken at once, and they hid behind each other.

First, the "test connection" button.It always probed /chat/completions.A TTS-only gateway exposes /models and /audio/speech, but no chat endpoint. So a perfectly healthy server was reported as broken.

Second, playback.Kokoro defaults to stream: true. The response was a chunked stream containing several complete WAV files glued together. And the blob URL on the frontend was built without the upstream Content-Type.

Simple fix. Set the header, done.Not quite.

The naive version would be "just add a MIME type." But a file with multiple RIFF headers back to back is not one valid WAV, no matter what label you put on it.

So the fix is layered:

→ Connection test probes /models, then /audio/speech, then /chat/completions→ The external TTS proxy sends stream: false, so the upstream returns one complete file→ As a defense for servers that still return concatenated WAVs, the backend merges the segments into a single WAV→ That merge has caps: a maximum total size (64 MB) and a maximum segment count (1,024), and it rejects anything whose RIFF headers don't add up→ The frontend builds blob URLs from the response Content-Type→ Only wav is supported on this path, and other formats are rejected with a clear message

The caps matter. Anything that parses untrusted bytes from a remote server needs bounds before it needs cleverness.

Verification: I added backend tests for the probe order, stream: false and the WAV merge, plus frontend tests for blob MIME handling. The PR body lists the relevant backend and frontend suites passing.

What I learned:

1. HTTP 200 only means the server replied.Whether the payload is playable is a separate question.

2. Health checks should match what the service actually offers.Probing one endpoint assumes every provider is a chat provider.

3. Defaults differ across "compatible" APIs.OpenAI-compatible rarely means identical. stream: true was a default I did not choose.

4. Metadata is part of the data.Losing a Content-Type turned valid audio into a failed media load.

5. Bound every parser that touches remote bytes.Size and segment limits are cheap insurance.

Compatibility is a spectrum, not a checkbox.

#Unsloth #BackendEngineering #TextToSpeech #Python #TypeScript

https://github.com/unslothai/unsloth/pull/9905

#47 LlamaParse and Unstructured in Dify's workerlanggenius/dify-official-plugins#3756

The plugin said it succeeded. The output was an error message.

That's the worst kind of failure in a Dify RAG pipeline: it looks like data, so the next node happily indexes it.

I ran into this in the Dify LlamaParse plugin when a knowledge pipeline passed in multiple files.

Two different errors showed up:A type confusion complaint about a nested list.And, in the Unstructured Transform plugin: "asyncio.run() cannot be called from a running event loop".

Same family, different symptoms.

Quick fix. Wrap the call and move on.That would have been the wrong fix.

First problem: what is File.blob, really?LlamaParse expects bytes. Dify can hand over a bytearray or a memoryview. With multiple files, the payload could end up as a nested structure. The library then fails far from the cause.

Second problem: Dify's plugin worker already runs inside an event loop (gevent-based), and these libraries call asyncio.run internally. That can't nest.

Third problem, and the one I found most important: errors were yielded as normal text output. A failed parse looked like a successful parse.

So the fix has a few parts:

→ Coerce each File.blob to bytes, including bytearray and memoryview→ Reject empty files, nested-list payloads and files without an extension-bearing name, with explicit errors→ Normalize a single file into a list so multi-file source nodes keep working→ Run load_data through a helper that applies nest_asyncio, so it works inside the running loop→ Apply the same approach to Unstructured Transform through a shared coroutine runner→ Raise parse and download errors instead of returning them as results

I added unit tests for the coercion cases, the rejection cases, and a call made from inside a running event loop.

What I learned:

1. A failure that returns text is a silent failure.In a pipeline, the only honest error is one that stops the node.

2. Types at a boundary are a contract.If a blob can be bytes, bytearray or memoryview, normalize it once, at the edge.

3. Know the runtime you're deployed into.asyncio.run works in a script and breaks in a worker that already owns the loop.

4. Multi-input paths need their own tests.The single-file case passed. The multi-file case is where it broke.

5. Make invalid input loud and early.A clear ValueError beats a stack trace from three layers down.

In RAG, bad parsing doesn't crash. It quietly poisons your index.

#Dify #AIEngineering #RAG #Python #Asyncio

https://github.com/langgenius/dify-official-plugins/pull/3756

#48 Dify DELETE body stripped by proxieslanggenius/dify#42162

I couldn't delete a custom model provider in Dify.

The button worked. The request went out. Dify's backend API replied with validation errors about missing fields.

But the fields were there. I could see them in the browser.

The cause was one line of HTTP semantics.

The web UI sent model, model_type and, for credential removal, credential_id in the JSON body of a DELETE request.

Many reverse proxies and gateways strip DELETE bodies. So the backend received an empty object, and Pydantic did exactly what it should: reject it.

Easy fix. Make the backend lenient.Except it didn't need to be.

The backend already accepted these fields as query parameters through validate_request(). The fix was on the client, not the server.

So the change was small and deliberate:

→ useDeleteModel and useDeleteModelCredential now send params instead of a body→ Both DELETE routes are documented in OpenAPI as query-parameter endpoints→ The generated contract types were updated to match→ Regression tests call the model and credential DELETE routes with a query string

No breaking API signature change. No new server behavior. Just aligning the client with what the server already supported.

The test suite for that controller passes, per the PR.

What I learned:

1. A request body on DELETE is a gamble.The spec leaves its meaning undefined, and intermediaries are free to drop it. Identify the resource in the URL.

2. The bug can be in the layer that works in dev.Locally there's no proxy. In a real deployment there is, and it changes what your API sees.

3. Check what the other side already supports before changing it.The smallest correct fix was already half-built.

4. Document what you rely on.Putting query parameters in the OpenAPI spec makes the contract explicit.

5. Test the transport shape, not only the handler.A test that sends the request the way a gateway would deliver it is worth more than a test that calls the function directly.

Some of the most frustrating production bugs aren't in your code. They're in what happens to your request on the way there.

#Dify #BackendEngineering #APIDesign #TypeScript #Python

https://github.com/langgenius/dify/pull/42162

#49 Symlink aliases and per-model settingsunslothai/unsloth#10644

I saved settings for a model. After a reload, they were gone.

Or worse: I made two symlinks to one model so each could have its own settings, and the model list showed only one of them.

This was in Unsloth Studio, in the backend code behind its local model inventory.

The cause was a deduplication rule. The inventory identified models by physical on-disk identity, the inode. Two symlinks to the same folder have the same inode, so they were collapsed into one row.

That made sense for avoiding duplicates. It didn't make sense for "Remember settings for this model", which is keyed by the path shown in the UI.

The obvious fix. Stop deduplicating.The obvious fix had a catch.

Without dedup, you get real duplicates in cases we do want to merge. So the rules needed to stay precise:

→ A symlink plus its canonical path stays one row, and the real path wins→ A loose file plus a symlink to the same file stays one row→ GGUF and safetensors at the same path stay two rows→ Overlapping scan roots, like a symlinked folder plus the real folder, stay one row→ Two distinct symlinks in a scan folder pointing at one model now stay two rows

The key idea: identity for the user is the path they see, not the inode underneath. So rows are now keyed by an alias path, with only the scan root resolved. Two registered roots reaching the same folder are one alias. Links below that root are not.

Then Windows.Path.is_symlink() returns False for a directory junction, and mklink /J needs no admin rights. So detection also reads the reparse tag for mount points.

Tests cover two symlink aliases staying distinct, two junction aliases on Windows, overlapping scan roots collapsing to one row, and a frontend test that alias paths keep independent remembered settings.

What I learned:

1. Identity depends on who is asking.The filesystem sees one model. The user sees two names with two configurations.

2. Dedup rules need explicit exceptions, not removal.Dropping the rule would have fixed one case and broken four.

3. Resolve as little as you need.Resolving the entire path erases the very distinction you want to keep.

4. Windows has its own link types.If your code only checks for symlinks, junctions will slip through.

5. Match the key to the feature.Settings keyed by path need rows keyed by path.

Deduplication is only correct relative to what the user considers the same thing.

#Unsloth #BackendEngineering #Python #TypeScript #LLM

https://github.com/unslothai/unsloth/pull/10644

#50 The autotune key nobody readunslothai/unsloth-zoo#1195

Nothing was wrong with the GPU kernels.The CPU was building a key nobody ever used.

This started from a CPU overhead report in Unsloth, the LLM fine-tuning library: about 2,500 fla launches per step on Qwen3.5-9B LoRA.

At that count, even tiny per-launch costs stop being tiny.

So I looked at the vendored fla CachedAutotuner.run.

On every Triton launch it built an AutotuneKey.Then Autotuner.run built its own key again.

The first one is only useful for a disk-cache lookup.But FLA_CACHE_MODE defaults to disabled.In that mode, no disk-cache lookup happens.

So the default path paid for a key it never read.

The fix was small:→ build the key only for modes that actually read the cache (default, always, strict)→ launches still go through Autotuner.run unchanged

What I measured, as stated in the PR:→ AutotuneKey.build with a gated-delta style signature (13 args, 7 tensors): about 4 us per call on CPU, now skipped per launch in the default mode→ trivial autotuned kernel launch, base vs head on the same GPU: roughly 31 us vs 25 us per launch (shared host, so indicative only)

I added a test that checks, for each FLA_CACHE_MODE, whether the key is built and the disk cache consulted, and that Autotuner.run still receives the original args. The disabled case fails on main.

What I took away:

1. Hot paths hide in plain sight.A few microseconds is invisible once. It is not invisible thousands of times per step.

2. Defaults deserve the most scrutiny.The default mode is what most users actually run.

3. Do work only when something will read it.If a value is computed for a feature that is off, it is pure overhead.

In LLM training, the cheapest operation is the one you never run.

#Unsloth #AIEngineering #Triton #Python #LLM

https://github.com/unslothai/unsloth-zoo/pull/1195

#51 Small Studio fixes that unblock usersunslothai/unsloth#10848, #10849, #9770

Three small Unsloth Studio bugs, all in its local LLM serving backend.None of them crashed anything.All of them left a user stuck with no way forward.

1. The log users sent me was empty.

After a normal Studio run, the session log held no llama-server output at all.Every forwarded line was logged at DEBUG, and DEBUG only went to a separate llama log.So the file people sent for debugging had nothing useful in it.

The fix:→ llama.cpp lines tagged W or E, plus the readiness line, now go to INFO→ unprefixed continuation lines inherit their record's level, so prompt text never reaches INFO because of its wording→ writes from the stdout drain are guarded, so a closed stdout can't stall llama-server

On a GPU, Studio from main put 0 llama-server lines in the session log. The branch put the two warnings and "model loaded" there.

2. Large GGUF downloads failed in Desktop.

"The file is too large to be downloaded using the regular download method."Without hf_xet, huggingface_hub refuses any single blob over 50GB.GGUF-only installs use --no-deps, so hf-xet was never pulled in transitively.The fix was to declare hf-xet explicitly in the requirements files and the studio extra, with a regression test.

3. Thinking controls never reached Ollama.

enable_thinking and reasoning_effort were not being sent.Now they map onto Ollama's OpenAI-compatible reasoning_effort:→ enable_thinking false sends none→ enable_thinking true with no effort sends medium→ minimal maps to low, xhigh maps to max

What I took away:

1. Diagnostics are a feature.If users send you a log, that log has to contain the failure.

2. Transitive dependencies are not a guarantee.A flag like --no-deps changes what "installed" means.

3. Pass-through settings need an explicit mapping.If a control isn't translated for a provider, it silently does nothing.

The smallest fixes often unblock the most people.

#UnslothAI #AIEngineering #LlamaCpp #LLM #HuggingFace

https://github.com/unslothai/unsloth/pull/10848https://github.com/unslothai/unsloth/pull/10849https://github.com/unslothai/unsloth/pull/9770

#52 RAG chunk and ingestion contractsinfiniflow/ragflow#18889, #20265, #19478

In RAG, the retrieval model gets the blame.But a lot of bad retrieval starts earlier, in the chunk contract.

Three RAGFlow fixes in its RAG ingestion pipeline and chunk API, one theme: be explicit about what a chunk is.

1. You could only append images.

The chunk PATCH API accepted image_base64.But if the chunk already had an image, the new one was always appended vertically.External ingestion tools couldn't replace or remove an image without deleting and reparsing the document.

I added an optional image_update_mode:→ append (default, current behavior)→ replace→ remove, which deletes the stored image and resets doc_type_kwd to text

2. Tables and images lost their surrounding text.

In the Go ingestion pipeline, GeneralChunker attached surrounding context for PDF, DOCX and spreadsheet paths.The markdown and HTML/plain-text paths never did.Markdown table units also never got a CKType, so the context hook would have skipped them anyway.The fix calls attachGeneralMediaContext on those paths and sets CKType on markdown table and image units.

3. A legacy field leaked into the Go ingestion DSL.

content_with_weight was still showing up before persistence.I made text the sole canonical chunk field until the final persistence rename:→ chunkers, tokenizer, extractor and tagger read text only→ the typed ContentWithWeight field is removed from ChunkDoc→ the persistence boundary requires a string text, then renames it to content_with_weight

What I took away:

1. Mutation APIs need more than "set".Append, replace and remove are different intents.

2. Every parser path needs the same enrichment.If one path skips it, retrieval quality varies by file type.

3. Pick one canonical field per stage.Two names for the same content means bugs at the boundary.

A chunk is a contract. Treat it like one.

#RAGFlow #BackendEngineering #RAG #Python #Golang

https://github.com/infiniflow/ragflow/pull/18889https://github.com/infiniflow/ragflow/pull/20265https://github.com/infiniflow/ragflow/pull/19478

#53 Dify plugin runtime and SDK compatibilitylanggenius/dify-plugin-daemon#819 + langgenius/dify#42481, langgenius/dify-plugin-sdks#385

Two Dify plugin bugs.Neither was in the plugin code.Both were in the plumbing around it: the Dify plugin daemon (Go) and the plugin SDK (Python).

1. Permission denied (os error 13)

On Docker with bind mounts, a common setup on WSL, plugin dependency installs failed.The plugin daemon always gave uv a cache at PLUGIN_WORKING_PATH/.uv-cache.That path sits on the bind mount, where host and container ownership often don't match.So uv failed with Permission denied while renaming cache entries.

This needed a fix on both sides:→ dify-plugin-daemon: UV_CACHE_DIR is now an optional override (the default stays PLUGIN_WORKING_PATH/.uv-cache), and the directory is created before uv runs→ dify: the Docker env examples drop the global UV_CACHE_DIR defaults and document it as an optional UV_CACHE_DIR=/tmp/.uv-cache setting for the plugin daemon, for bind-mount permission errors

I added a unit test for the cache-dir resolution.The Dify change needs a matching daemon release to take full effect, so I said that in the PR.

2. Pydantic rejected a value Dify core itself emits.

In agent/chat mode, MCP blob resources produced this error:Input should be 'text', 'file', 'blob', 'json', 'link', 'image', 'image_link', ...input_value='binary_link'

Dify core's message_transformer emits type binary_link for non-image blobs.The SDK enum InvokeMessage.MessageType didn't have it.I added BINARY_LINK and tests that ToolInvokeMessage and DatasourceMessage accept it.

What I took away:

1. Runtime state belongs on container-local storage.Caches on bind mounts inherit host permission problems.

2. Fix the contract on both sides of a boundary.A daemon change and an env-example change were one bug.

3. Enums are wire contracts.If one side can emit a value, the other side's enum has to know it.

Many "plugin bugs" are really version-skew bugs in disguise.

#Dify #BackendEngineering #Docker #MCP #Golang

https://github.com/langgenius/dify-plugin-daemon/pull/819https://github.com/langgenius/dify/pull/42481https://github.com/langgenius/dify-plugin-sdks/pull/385

#54 Request and payload edge casesBerriAI/litellm#36513, langgenius/dify#41761

The happy path worked in both of these.In LiteLLM's LLM API gateway and Dify's backend, the same data in a slightly different shape did not.

1. A video edit returned 500 on form input.

Through the LiteLLM proxy, JSON requests to /v1/videos/edits worked.Form-encoded and multipart requests returned 500.The OpenAI SDK's client.videos.edit() failed with an internal server error.The proxy log showed: RuntimeError: Stream consumed.

The cause: the handler read the raw request body again, after auth had already consumed the stream.The fix reads through the same cached post-auth body path used elsewhere.It also normalizes the video field into video_id, whether it arrives as a dict, a bare id string, or a JSON-encoded string.Video remix got the same body handling.

I checked it by running the same requests against a proxy before and after the change. Form-urlencoded and multipart edits went from 500 to 200.

2. A text file was reported as binary.

In Dify's binding-file preview, the file was hard-sliced at max_bytes.Then it was decoded as strict UTF-8.If the cut landed inside a multibyte character, the decode failed and the whole file was called binary.

For example, a file containing a€ with max_bytes 2 cuts the euro sign in half.

The fix keeps the longest valid prefix, but only when the error is "unexpected end of data" at the end of the buffer.Invalid bytes in the middle still classify the file as binary.Tests cover 2-, 3- and 4-byte splits.

What I took away:

1. A body is a stream, not a variable.Read it once, then reuse the cached copy.

2. Same data, different encoding.JSON, form and multipart should all reach the same handler logic.

3. Truncation can create the error you then report.Distinguish "cut off" from "actually invalid".

Most edge cases aren't exotic. They're the same input in a different shape.

#LiteLLM #Dify #BackendEngineering #FastAPI #LLM

https://github.com/BerriAI/litellm/pull/36513https://github.com/langgenius/dify/pull/41761

View all merged PRs →