Adds snapshot testing to the built-in test runner: a new
`t.assertSnapshot(actual, options?)` method on `Deno.TestContext` and an
`--update-snapshots` (`-u`) flag for `deno test`. Snapshots are stored
in the same `__snapshots__/<test file>.snap` format as
`@std/testing/snapshot` (same serializer settings), so existing snapshot files keep working
after switching to the built-in API.
Compared to the userland implementation, update mode is a proper CLI
flag instead of `deno test -- --update`, and no read or write permissions are
needed for snapshot files in the default location since the runner
manages them itself (custom `dir`/`path` locations still require permissions,
and snapshot writes are only authorized when `--update-snapshots` is
passed).
Updated and removed snapshot counts are reported through the test
reporters in the run summary. Stale snapshots are only removed when every test in
a module ran all of its assertions, so filtered, ignored, `only` and
failed runs never delete snapshots they didn't see.
Out of scope for this PR, noted as follow-ups: inline snapshots
(`assertInlineSnapshot`), `expect(...).toMatchSnapshot()` on top of the
same machinery, Jest-style CI behavior (failing on newly written snapshots in
CI), and a deprecation pointer in `@std/testing/snapshot` once this
ships.
Migrating a test suite from a third-party runner like Vitest to `deno
test` is harder than it should be, partly because there's no built-in way to
tolerate or surface flaky tests. This adds two per-test options that
mirror Vitest's semantics: `retry` re-runs a failing test and passes if any
attempt passes, while `repeats` runs the test several times and requires every
run to pass. They compose, so each repetition may itself be retried. Matching
`--retry` and `--repeats` flags set a default for the whole run; a test
that sets its own option takes precedence, including an explicit `0` that
opts the test out of a flag-provided default.
The runner already drives one JS call per test from Rust, so this wraps
that call in an attempt/repetition loop. Each attempt re-runs the beforeEach
and afterEach hooks and captures a fresh sanitizer baseline, which keeps
every attempt's leak check independent and makes a resource leak retryable
like any other failure. A failed-but-retried attempt is surfaced through a new
informational event so the reporter can print it, and a test that
ultimately passes after a retry is counted as `flaky` in the summary line. Hook
errors and uncaught errors are intentionally not retried.
A couple of limitations worth calling out: there is no `deno.json`
configuration for retry/repeats yet (CLI flag and per-test option only);
and the `junit` and `tap` reporters don't yet treat retried attempts
specially, so a retried test re-registers its steps under fresh ids and emits
duplicate `<testcase>`/step entries in those reporters (the pretty and dot
reporters drop the discarded attempts). When a test with steps passes on a later
attempt, the final attempt's passing steps are counted but not
re-printed individually (the earlier failed attempt's steps are still shown). None
of these affect correctness of the pass/fail verdict or the flaky count.
Closes https://github.com/denoland/deno/issues/19882
Replaces the Rust-based ZMQ kernel implementation (using `zeromq`,
`runtimelib`/`jupyter_runtime`, and `jupyter-protocol` crates) with a
pure-JS ZMTP 3.1 implementation running on top of `Deno.listen()`.
**Architecture**
- **Main OS thread** — new `cli/js/jupyter_kernel.js` (~800 lines) owns
all ZMQ/Jupyter protocol work: ZMTP 3.1 framing, NULL handshake,
HMAC-SHA256 message signing, and all five channel loops (shell, control,
iopub, heartbeat, stdin). Uses only non-blocking `Deno.listen()` TCP, so
the event loop never blocks.
- **Background OS thread** — runs the REPL session (user code
evaluation, CDP, TypeScript). Created via a cloned
`CliMainWorkerFactory`.
- The two threads talk over `tokio::mpsc` channels. The main thread can
interrupt user code at any time by calling
`IsolateHandle::terminate_execution()` on the handle sent back from the
background thread via a `oneshot` channel.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
## Summary
Calling `Deno.exit()` from outside any test function — at module top
level, in an `unload` listener, or from async work that escaped a test —
used to kill the entire `deno test` process with the requested exit
code. That silently dropped buffered test results and made failing tests
"pass" with exit code 0.
Reproducer from #12888:
```ts
Deno.test("fail", () => { throw new Error("fail"); });
self.onunload = () => { Deno.exit(0); };
```
Before this PR:
```
$ deno test test.ts
Check file:///.../test.ts
running 1 test from file:///.../test.ts
test fail ... FAILED (9ms)
$ echo $?
0 # failed test, but zero exit code
```
After this PR:
```
$ deno test test.ts
running 1 test from ./test.ts
fail ... FAILED (1ms)
note ./test.ts called `Deno.exit(0)` from outside any test. The isolate was terminated; remaining test files will continue.
ERRORS
fail => ./test.ts:1:6
error: Error: fail
...
FAILED | 0 passed | 1 failed (24ms)
error: Test failed
$ echo $?
1
```
## How it works
Each test specifier now installs a default exit handler (in addition to
the per-test `assertExit` handlers) that, when `Deno.exit(code)` is
called outside of any test function, calls a new `op_test_isolate_exit`
op:
- records the exit code in `OpState` (`IsolateExitInfo`)
- sends a new `TestEvent::IsolateExit(origin, exit_code)` to the
reporter
- asks V8 to terminate execution of just this isolate (uncatchable)
The test runner detects the termination via the `IsolateExitInfo` flag,
calls `cancel_terminate_execution`, and continues with any remaining
specifiers. A zero exit code is treated as a clean isolate teardown; a
non-zero exit code fails the run.
The existing `sanitizeExit: false` path (which kills the whole process
via `std::process::exit`) is preserved for `Deno.exit()` calls from
inside a test function — only the *outside-a-test* path changes here,
since that's where the silent-process-kill bug lived.
Fixes#12888.
Closesdenoland/orchid#325
## Test plan
- [x] Existing `exit_sanitizer`, `exit_sanitizer_disabled`,
`exit_code{,2,3}`, `before_unload_prevent_default` spec tests still pass
- [x] New `exit_in_isolate_unload` spec test (the literal #12888
reproducer): test fails, isolate-exit note is printed, process exits 1
- [x] New `exit_in_isolate_top_level::exit_zero_alone`: top-level
`Deno.exit(0)` -> exit 0, no phantom tests
- [x] New `exit_in_isolate_top_level::exit_nonzero_alone`: top-level
`Deno.exit(7)` -> exit 1, failure reported
- [x] New
`exit_in_isolate_top_level::mixed_run_passing_alongside_isolate_exit`: a
passing test file run alongside an isolate-exit file still completes
normally
- [x] `cargo clippy --bin deno` clean
---------
Co-authored-by: divybot <divybot@users.noreply.github.com>
Co-authored-by: Divy Srivastava <me@littledivy.com>
## Summary
Follow-up to #22515 / fixes#22516.
Previously, when a test set `sanitizeExit: false`, we treated the exit
sanitizer as completely disabled — a test calling `Deno.exit()`
terminated the process immediately. Because the test reporter's
stdout/stderr is buffered and flushed asynchronously through a channel,
this could drop output and produce flaky test runs. #22515 had to
comment out a node-compat test (`should allow exit`) precisely because
of this unreliable flushing.
This PR makes the exit handler always present during tests, with
behavior depending on `sanitizeExit`:
- **`sanitizeExit: true` (default)** — unchanged. An exit attempt fails
the current test (and a non-zero exit code set during a test fails it
too), and the remaining tests keep running.
- **`sanitizeExit: false`** — instead of silently terminating the
process, the test run is **aborted with a message**. The worker hands
the exit code to the reporter, which prints a message, reliably flushes
all buffered output, and then exits the process with the requested code.
Example output:
```
running 2 tests from ./main.ts
exit ...
------- output -------
output before exit
----- output end -----
warning A test called Deno.exit(42) while the exit sanitizer was disabled (sanitizeExit: false). Aborting the test run.
The following tests were pending:
exit => ./main.ts:1:6
```
(process exits with code 42)
## Implementation
- New `op_test_event_exit` op sends a new `TestEvent::Exit(code)` to the
reporter (which forces a stdio sync/flush first), then parks the worker
so user code can't keep running while the process is torn down.
- `report_tests` handles `TestEvent::Exit` much like
`TestEvent::Sigint`: it reports the pending tests, flushes the report,
and exits the process with the given code. A `report_exit` method was
added to the `TestReporter` trait (pretty/dot/tap/junit/compound).
- `op_test_event_exit` is added to the `NOT_IMPORTED_OPS` allowlist in
`99_main.js` so it stays available on `Deno.core.ops` (where the other
`deno_test` ops live).
## Tests
- New spec test `test/exit_sanitizer_disabled` covering the
abort-with-message + reliable-flush behavior and exit code propagation.
- Re-enables the `should allow exit` node-compat test that #22515
disabled, updating its expected output and exit code (123).
- Verified the default `sanitizeExit: true` behavior
(`test/exit_sanitizer`) is unchanged.
Closesdenoland/orchid#282
---------
Co-authored-by: divybot <divybot@users.noreply.github.com>
Co-authored-by: Divy Srivastava <me@littledivy.com>
Closes#11133.
Closes#13146.
## Summary
Adds a per-test `timeout` option to `Deno.test()` that fails a
test if it runs longer than the configured deadline. Both
asynchronous hangs and synchronous hot loops are caught:
```ts
Deno.test({
name: "fetches within deadline",
timeout: 5000,
async fn() {
await fetch("https://example.test");
},
});
```
If `timeout` is unset or `0`, the test runs without a deadline.
Upgrading jupyter dependencies. After this one is merged I'll be able to
refactor the jupyter code to use the new `.split()` available on the
sockets to make the code cleaner. I think it also opens up some things
we couldn't handle before.
1. Enforces using `resolve_cwd`, which is more efficient and reduces.
2. Uses `canonicalize_path` instead of `.canonicalize()` because it
strips the unc path on Windows without us having to remember to do that.
Fixes#30787.
`Deno.test()` was being accepted even when called from inside a running
test callback.
That means nested registrations could silently be ignored and tests
looked like they passed.
Now `Deno.test()` calls made from inside a running test are rejected
immediately.
Instead of silently registering too late, they fail with a clear error:
`Nested Deno.test() calls are not supported. Use t.step() for nested
tests.`
This makes the misuse explicit and points users to `t.step()`.
In anticipation for #29357 to reduce the diff & complexity in that PR.
Reason for this is that deno_canvas will include the actual canvas
implementation, and both deno_webgpu & deno_canvas need to depend on
deno_image, so this cannot be merged into deno_canvas for now.
This commit adds `Deno.test.beforeAll`, `Deno.test.beforeEach`,
`Deno.test.afterAll` and `Deno.test.afterEach` APIs.
These APIs can be used to perform setup and teardown for test cases.
The API is similar to the Vitest API: https://vitest.dev/api/#setup-and-teardown,
with the main difference being that that `before*` hooks don't return a cleanup
function.
Since `rust 1.87.0` reported `undefined symbol:
ring::pbkdf2::PBKDF2_HMAC_SHA1::*` in CI and it was difficult to debug
locally, use `rust 1.86.0` in CI tests for troubleshoot the errors
This commit removes the docs URL for diagnostics coming
from JS plugins. Before, we mistakenly printed
URLs pointing to `docs.deno.com`, even though they did not
exist.
An ability to actually specify a custom URL for plugin rules,
will be added in a follow up PR.
This clears the diagnostics whenever a file is about to run. For
example, what could previously occur is an error and the diagnostics
would be leftover from the previous run.
This commit adds an unstable lint plugin API.
Plugins are specified in the `deno.json` file under
`lint.plugins` option like so:
```
{
"lint": {
"plugins": [
"./plugins/my-plugin.ts",
"jsr:@deno/lint-plugin1",
"npm:@deno/lint-plugin2"
]
}
}
```
The API is considered unstable and might be subject
to changes in the future.
Plugin API was modelled after ESLint API for the
most part, but there are no guarantees for compatibility.
The AST format exposed to plugins is closely modelled
after the AST that `typescript-eslint` uses.
Lint plugins use the visitor pattern and can add
diagnostics like so:
```
export default {
name: "lint-plugin",
rules: {
"plugin-rule": {
create(context) {
return {
Identifier(node) {
if (node.name === "a") {
context.report({
node,
message: "should be b",
fix(fixer) {
return fixer.replaceText(node, "_b");
},
});
}
},
};
},
},
},
} satisfies Deno.lint.Plugin;
```
Besides reporting errors (diagnostics) plugins can provide
automatic fixes that use text replacement to apply changes.
---------
Co-authored-by: Marvin Hagemeister <marvin@deno.com>
Co-authored-by: David Sherret <dsherret@gmail.com>
This PR extracts the core part of
https://github.com/denoland/deno/pull/27203 to make it easier to review
and land in parts.
It contains:
- The JS plugin code the deserializes and walks the buffer
- The Rust portion to serialize SWC to the buffer format (a bunch of
nodes are still todos, but imo these can land anytime later)
- Basic lint plugin types, without the AST node types to make this PR
easier to review
- Added more code comments to explain the format etc.
More fixes and changes will be done in follow-up PRs.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
This commit improves permission prompts by adding an option
to print a full trace of where the permissions is being requested.
Due to big performance hint of stack trace collection, this is only
enabled when `DENO_TRACE_PERMISSIONS` env var is present.
Closes https://github.com/denoland/deno/issues/20756
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
`performance.timeOrigin` was being set from when JS started executing,
but `op_now` measures from an `std::time::Instant` stored in `OpState`,
which is created at a completely different time. This caused
`performance.timeOrigin` to be very incorrect. This PR corrects the
origin and also cleans up some of the timer code.
Compared to `Date.now()`, `performance`'s time origin is now
consistently within 5us (0.005ms) of system time.
Closes: https://github.com/denoland/deno/issues/22633
This commit adds support for `confirm` and `prompt` APIs,
that instead of reading from stdin are using notebook frontend
to show modal boxes and wait for answers.
---------
Co-authored-by: Bartek Iwańczuk <biwanczuk@gmail.com>
Also removes permissions being passed in for node resolution. It was
completely useless because we only checked it for reading package.json
files, but Deno reading package.json files for resolution is perfectly
fine.
My guess is this is also a perf improvement because Deno is doing less
work.
This brings in [`runtimelib`](https://github.com/runtimed/runtimed) to
use:
## Fully typed structs for Jupyter Messages
```rust
let msg = connection.read().await?;
self
.send_iopub(
runtimelib::Status::busy().as_child_of(msg),
)
.await?;
```
## Jupyter paths
Jupyter paths are implemented in Rust, allowing the Deno kernel to be
installed completely via Deno without a requirement on Python or
Jupyter. Deno users will be able to install and use the kernel with just
VS Code or other editors that support Jupyter.
```rust
pub fn status() -> Result<(), AnyError> {
let user_data_dir = user_data_dir()?;
let kernel_spec_dir_path = user_data_dir.join("kernels").join("deno");
let kernel_spec_path = kernel_spec_dir_path.join("kernel.json");
if kernel_spec_path.exists() {
log::info!("✅ Deno kernel already installed");
Ok(())
} else {
log::warn!("ℹ️ Deno kernel is not yet installed, run `deno jupyter --install` to set it up");
Ok(())
}
}
```
Closes https://github.com/denoland/deno/issues/21619
1. Generally we should prefer to use the `log` crate.
2. I very often accidentally commit `eprintln`s.
When we should use `println` or `eprintln`, it's not too bad to be a bit
more verbose and ignore the lint rule.
Issue https://github.com/denoland/deno/issues/22222
This PR moves `runtime/permissions` code to a upstream crate called
`deno_permissions`. The `deno_permissions::PermissionsContainer` is put
into the OpState and can be used instead of the current trait-based
permissions system.
For this PR, I've migrated `deno_fetch` to the new crate but kept the
rest of the trait-based system as a wrapper of `deno_permissions` crate.
Doing the migration all at once is error prone and hard to review.
Comparing incremental compile times for `ext/fetch` on Mac M1:
| profile | `cargo build --bin deno` | `cargo plonk build --bin deno` |
| --------- | ------------- | ------------------- |
| `debug` | 20 s | 0.8s |
| `release` | 4 mins 12 s | 1.4s |
- Removes the origin call, since all origins are the same for an isolate
(ie: the main module)
- Collects the `TestDescription`s and sends them all at the same time
inside of an Arc, allowing us to (later on) re-use these instead of
cloning.
Needs a follow-up pass to remove all the cloning, but that's a thread
that is pretty long to pull
---------
Signed-off-by: Matt Mastracci <matthew@mastracci.com>
Bartek Iwańczuk