Chat-O Home • Back to Blog

Benchmarking DeepSeek V4.1 Flash on a Real Upgrade: salad_ui 1.0 and Phoenix LiveView 1.2

Published:

Chat-O is an Elixir and Phoenix LiveView app. We upgraded Phoenix LiveView, which renders the interface, and salad_ui, the generator used for components that are vendored into the app. The upgrades were coupled: salad_ui 0.14.9 did not parse under the LiveView 1.2 tokenizer, which was the reason LiveView had been pinned to 1.1.x.

That made this a useful agent benchmark. The code change was small, but getting it right required dependency reasoning, production-build awareness, automated tests, and browser QA. We ran the task with DeepSeek V4.1 Flash through OpenCode Go in a single session.

The change

The dependency upgrade itself changed two files, with 17 insertions and 17 deletions:

 mix.exs  | 16 ++++++++--------
 mix.lock | 18 +++++++++---------
 2 files changed, 17 insertions(+), 17 deletions(-)

The meaningful part of mix.exs:

-      {:phoenix_live_view, "~> 1.1.0"},
+      {:phoenix_live_view, "~> 1.2"},

-      {:salad_ui, "~> 0.14"},
+      {:salad_ui, "~> 1.0", only: [:dev]},
+      {:tw_merge, "~> 0.1"},

-      {:igniter, "~> 0.5", only: [:dev]},
+      {:igniter, "~> 0.6", only: [:dev]},

The line worth highlighting is salad_ui. Version 1.0 depends on igniter, a code-generation toolkit. Chat-O does not call salad_ui at runtime because its generated components are vendored and use TwMerge directly. The agent therefore kept salad_ui and igniter development-only, then added tw_merge as a direct dependency.

That is more than dependency-file bookkeeping. It keeps salad_ui, igniter, and the rest of the code-generation toolchain out of the production dependency set while retaining the one runtime library the vendored components need.

Verification

Tests were the baseline, not the finish:

  • Development compilation succeeded.
  • Production compilation succeeded, with salad_ui and igniter absent from the production build and tw_merge present.
  • 401 of 401 tests pass.

Then it did the part we did not ask for: a manual QA pass in a real browser.

Flow Result
Login and session works, socket connected
Chat with image history renders, image messages intact
Image Studio modal opens; thumbnails; main image loads; edit box present
Thumbnail click switches versions
Arrow-key navigation moves between versions, with wrap
Escape closes the modal
Sidebar tabs panel switches
Prompts panel opens with the template list
Room picker shows the available personas
Composer accepts and holds typed text
Settings renders, connected
Blog post images load, related posts present

The checked flows produced no console errors, broken images, or reconnect banners.

OpenCode Go running DeepSeek V4.1 Flash: the verification table and run stats, 10.5s at 174.5 tokens per second

The screenshot reports 10.5 seconds and about 174 tokens per second for the displayed model turn. That is a single observed turn in our setup, not the wall-clock duration of the full upgrade, compilation, test, and browser-QA session. It is useful as a glimpse of generation speed, not as a standalone end-to-end benchmark.

Why the manual pass mattered

The 401-test automated suite passed, but it does not drive every browser interaction or render every state of every vendored component. In particular, it does not open the Image Studio modal and use its keyboard navigation. That is exactly the kind of surface a UI framework upgrade can break while an otherwise healthy suite stays green.

The browser pass complemented the automated suite. It opened the app, exercised the named flows in the table, and checked observable browser behavior: image loading, console output, and LiveView connectivity. That gave us useful coverage of several interactions the existing suite does not drive.

This distinction matters. Automated tests, targeted regression tests, and browser walkthroughs catch different classes of problems. A good agent run should use them together and report the boundaries of each, not treat one as a substitute for the others.

It even had to unstick its own tooling. A previous automated browser was holding a profile lock and blocking a fresh launch, so the agent found the process, cleared it, and relaunched.

Where this lands

It looks like a clean upgrade. The dependency decision is technically sound, the existing suite passed, the production build excluded the development-only tooling as intended, and the browser pass covered several high-value interactions outside that suite.

The result is also a good reminder of what a benchmark should show. The interesting part was not that a model edited two files. It was that the model found the dependency boundary, preserved the production build, verified the application at multiple levels, and documented what it observed. We still review that work like any other change before it goes to production.

For more of our agent-driven engineering, see how we migrated a full-stack codebase from Rails to Go with a model most people had not heard of.

You may also like