Deduplicate exact and near-matching questions before assessing generalization.
Research case study · 13 OCT 2026
Qwen 27B Domain QLoRA
Extended Evaluation
A candid look at domain adaptation, tool-aware quality, speculative inference, and the latency costs of concurrent local-LLM serving on a single NVIDIA DGX Spark.
1 × DGX Spark · 128 GB unifiedSGLang + DFlash2v0.5 · Internal trial candidateArchived evidence · Not rerun
Read the evidence boundary. Results reflect an application-level assistant (prompts, retrieval and tools), not isolated LoRA quality. The 39- and 100-case grades are single-reviewer observations, not independent accuracy scores; the public dataset contains aggregate values only. No production-capacity or production-release claim is made.
Key figures
ACCEPTANCE SUITE
37/39
37 pass · 2 partial · 0 fail
DEVELOPMENT SUITE
91/100
91 pass · 4 partial · 5 fail
8-STREAM AGGREGATE
110.8 tok/s
Fixed 512-token output · not single-user
LONG INPUT PROBE
171k tokens
Needle retrieved · 178.6s TTFT
01. Quality, with boundaries
Completion and reviewer grading are different things. We show PASS / PARTIAL / FAIL rather than overstating the results as model accuracy.
Archived result classifications
Small, domain-oriented internal suites; a single AI-assisted reviewer graded the two larger suites.
PASSPARTIALFAIL
Not a held-out accuracy score. Development items are near-distribution, and the final 134-question holdout was not evaluated. Five holdout questions have known verbatim training overlap.
Failure concentration in tool-assisted cases
The model sometimes allowed retrieved records to override defined answer constraints.
2 non-PASS70 PASS · 1 PARTIAL · 1 FAIL
7 non-PASS21 PASS · 3 PARTIAL · 4 FAIL
Engineering finding: tool outputs need policy-boundary checks, not just correct tool selection. Different question difficulty prevents a causal interpretation of this split.
Fixed gates Knowledge 5/5 · Tool workflow 6/6 · In-depth reports completed 8/8 (content accuracy not asserted).
02. Throughput is not responsiveness
At higher concurrency, combined throughput improves while some requests wait much longer for their first token.
Fixed-output concurrency sweep
512 generated tokens/request · greedy · reasoning OFF · one GB10 · no application tool calls.
8 streams: 110.8 aggregate tok/s8 streams: 0.30s worst TTFT32 streams: 83.32s worst TTFT
The 32-stream 145.7 tok/s value is the sum across all requests. Individual-request averages, queueing time and tail delay must be reported alongside total output. This workload cannot establish a safe number of concurrent production users.
Streaming gate: two repeated observations
Original fixed speed-gate construction, measured as output decode tokens/s and **excluding TTFT**. Different protocol from the concurrency sweep.
39.93 tok/s
38.95 tok/s
These are not evidence that LoRA or DFlash2 alone caused a speedup; the training, quantization, runtime and decoder are a combined serving configuration.
Long context: one needle, rising cost
One synthetic retrieval probe per context length; the needle was recovered on all seven runs.
170,979 input tokens → 178.61 seconds TTFT. This measures a simple lookup with one probe per length, not multi-document comprehension or long-context reasoning accuracy.
03. The unresolved engineering work
Deliberately including failures is part of making this study useful to other engineers.
01 / POLICY BOUNDARY
Retrieved data can override definitions
Some tool-assisted development cases treated observed records as permission to change a canonical rule. Separate tool accuracy from answer-policy adherence.
02 / OUTPUT SAFETY
Identifier masking needs its own gate
A recorded response exposed internal identifiers where a masked or synthetic example was expected. No values are included in this publication.
03 / MODEL SERVING
Thinking without a visible answer
Some nonstandard prompt conditions exhausted tokens in reasoning or produced an empty visible-answer field. Parser and end-of-answer checks are essential.
04 / EVALUATION
Independent holdout not established
The final 134-case set was not scored and has five known verbatim overlaps with training text. It must be cleaned before independent evaluation.
05 / USER EXPERIENCE
Long answers carry high latency
The fixed depth gate completed 8/8 reports but averaged 406.4 seconds each. Answer generation is not proof of answer correctness.
06 / PORTABILITY
Language and response-mode drift
Short-writing responses could become too terse when reasoning was off; some Chinese replies mixed character variants. Prompt and mode matter.
04. How to evaluate more rigorously
A proposed next protocol, not a claim that these steps were already completed.
Grade tool choice, final answer, policy boundaries and masking separately.
Measure TTFT, answer-vs-reasoning tokens, latency and tail behavior.
Log variance, cache state and reviewer disagreements rather than a single score.
Evidence provenance & disclosure
All numbers shown here were manually whitelisted from archived October 2026 internal evaluation artifacts and cross-checked against their summary records. This **new** visualization does not embed the original private dashboard or executable training code. The underlying prompts, answers, SQL, customer or member data, endpoints, deployment files and raw logs are deliberately absent. Historical engineering execution was AI-agent-assisted, with the project owner directing requirements and decisions.
Publication status: de-identified research summary published with project-owner approval; not an accepted Production service. This page is not evidence of a formally approved production service.