Data engineers often receive a file before they receive a clean problem statement.

Before building a pipeline, the useful questions are usually simpler:

  • What is the schema?
  • Did we detect the delimiter and header correctly?
  • Which columns contain nulls, mixed types, duplicates, or outliers?
  • Can this file be ingested safely?
  • What query produces the subset we actually need?

I built DuckPad as an engineering prototype around that first investigation.

Project page: DuckPad.

The central constraint is local-first: inspecting a file should not require making “upload the raw dataset to somebody else’s server” the default architecture.

The boundary that matters

The UI thread should render, accept user intent, expose progress, and remain cancellable. It should not parse gigabyte files or execute analytical queries.

The proposed runtime therefore separates responsibilities:

Local file
   ↓
Import coordinator
   ↓
Dedicated Web Worker
   ↓
DuckDB-Wasm
   ↕
OPFS temporary storage
   ↓
Arrow batches
   ↓
Virtualized profiling + query UI

The browser gives the application a File or file handle. A small sample is read first to detect format, encoding, delimiter, and possible headers. Ambiguous import settings should be confirmed by the user before expensive work starts.

Only then does the worker register the file with DuckDB-Wasm and execute preview or profiling queries.

Preview before import

One of the easiest mistakes in data tooling is to treat “open the file” as SELECT *.

For large data, the first useful screen rarely requires the full dataset. Preview queries can use:

  • projection;
  • predicates;
  • bounded row counts;
  • sampling;
  • schema queries;
  • targeted aggregate checks.

That reduces both latency and memory pressure while producing information that helps decide what to do next.

Keep expensive work away from the interface

The worker boundary exists for more than aesthetic smoothness.

Its responsibilities include:

  • DuckDB-Wasm initialization;
  • file registration;
  • SQL execution;
  • CSV dialect sampling;
  • schema inference;
  • profiling and quality checks;
  • cancellation and timeouts.

The main thread stays responsible for accessibility, navigation, controls, progress, and bounded rendering.

If a query runs for seconds, the UI still needs to acknowledge cancellation immediately even if the analytical engine needs a moment to stop.

Arrow instead of giant serialized tables

For analytical results, sending a huge array of JavaScript objects across the worker boundary is a poor default.

The architecture instead assumes Arrow-oriented batches that can be streamed or transferred efficiently to a virtualized table.

The UI should have its own visible row limit independently of what the query engine is capable of computing.

This distinction matters: query capacity and rendering capacity are different budgets.

Memory is a product constraint

A browser is not a magic data warehouse.

The memory strategy I documented for DuckPad is deliberately defensive:

  1. sample before full import;
  2. estimate projected memory;
  3. prefer column projection and predicate pushdown;
  4. paginate or stream results;
  5. use OPFS for temporary database state where supported;
  6. set UI and query limits independently;
  7. provide cancellation;
  8. document when a desktop or cloud execution path is the honest fallback.

Local-first should not become “pretend every dataset fits in a browser”.

Performance needs named metrics

“Fast” is too vague to be useful.

For DuckPad, the benchmark plan separates:

  • time to first preview;
  • time to stable schema;
  • import throughput;
  • query latency at p50 and p95;
  • peak memory;
  • main-thread responsiveness;
  • export throughput;
  • cold and warm Wasm startup.

The initial UX budgets are similarly explicit: controls should respond within one animation frame, intentional main-thread tasks should stay below 50 ms, progress should appear quickly for work expected to exceed a second, and previews should remain bounded.

Those are engineering targets, not marketing guarantees.

A prototype should say what is not built

The current repository contains a polished landing page, a standalone workspace prototype, testable CSV-detection logic, CI/CD structure, and documentation for architecture, testing, security, observability, performance, and error handling.

The full DuckDB-Wasm execution path, Parquet path, OPFS persistence, and real cloud connectors are still roadmap items.

That distinction is important to me. A credible engineering project should make it easy to separate:

  • implemented capability;
  • proposed architecture;
  • measured behavior;
  • future work.

Otherwise a portfolio prototype becomes a collection of claims nobody can audit.

When a server becomes justified

A server is not required for the local inspection problem itself.

It becomes useful when the product grows into things such as:

  • billing and team accounts;
  • encrypted synchronization of rules;
  • shared projects and comments;
  • scheduled jobs;
  • centralized policy administration;
  • datasets larger than the browser execution envelope.

The useful idea is not “everything must stay client-side forever”. It is put the service boundary where the product actually needs one.

That is the larger reason I built DuckPad: not only to prototype a data tool, but to make the architectural trade-offs visible enough to discuss, test, and improve.