aphrody-labs/wordcraft
1
0 Fork 0
mirror of https://github.com/aphrody-labs/wordcraft.git synced 2026-10-13 19:26:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

M15: Polish interface, and Polish spelling, grammar and hyphenation (#346)

Browse Source 
* M15: Polish interface, and Polish spelling, grammar and hyphenation

Interface
- Polish is the tenth interface language ("Polski", pl.tsv, 1,045 entries);
  pl, pl-PL and pl_PL.UTF-8 system locales select it automatically.
- Right-click menu labels now go through translation in every language.
- Review > Language is a menu (English US/UK, Polski, Do not check spelling
  or grammar); the status bar shows the language at the caret.
- With a Polish interface, new blank documents and opened .txt/.md files
  default to pl-PL.

Proofing
- Each run is checked in its own language (w:lang, resolved through
  styles): squiggles, F7, review.issues, suggestions and hyphenation.
  Text in languages without proofing tools is no longer flagged as English.
- Polish spelling from the SJP.PL Hunspell dictionary (Apache-2.0 option),
  read by a new Rust affix engine (crates/proof/src/affix.rs); 875 KB asset,
  ~4.5M word forms. Regenerate with the pl_dictionary example.
- Polish suggestions: case, missing diacritics, rz/ż, ó/u, ch/h confusions,
  missing spaces (napewno -> na pewno), edit distance.
- Conservative Polish grammar rules: repeated words, spacing, doubled
  punctuation, sentence capitals, missing comma before że/który/ponieważ/
  aby/żeby/jeśli…, we/ze forms.
- Polish hyphenation with the hyph-utf8 hyph-pl.tex patterns (MIT option);
  pattern keys widened to 20 letters, TeX exceptions parsed; the heuristic
  no longer treats ł, ż, ś… as vowels.

review.language validates and normalizes tags, can set noProof alone and
the document default; format.state reports lang and noProof.

Tests for every part, docs and ATTRIBUTION.md/NOTICE updated;
cargo xtask ci passes (fmt, clippy, test, assets, layers, wasm).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Layout: rustfmt the test module order left unformatted by #391

cargo fmt --check failed on main after #391 added mod tests_grid.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Review fixes for #346: read Polish spelling from the installed Hunspell dictionary

- Polish spelling uses the Hunspell pl_PL dictionary installed on the
  system (OS packages, Homebrew, LibreOffice, ~/.local/share/hunspell, or
  WORDCRAFT_DICTIONARIES) instead of a bundled 870 KB SJP.PL file;
  without one, Polish text gets no spelling marks, like Word without the
  language's proofing tools. .aff/.dic are size-capped and decoded from
  UTF-8 or ISO 8859-2 (crates/proof/src/hunspell.rs).
- The dictionary can be replaced at runtime (install_dictionary) or
  re-searched (reload_dictionary); an empty one accepts every word.
- Tests use small synthetic dictionaries written for them, so they don't
  depend on what the machine has installed.
- The bundled pl.dic, its ATTRIBUTION/NOTICE entries and the generator
  example are removed; docs describe the installed dictionary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* M15: Polish catalog covers the interface text merged from main

The merge of main (98456ac) brought ~575 new interface strings (charts,
SmartArt, content controls, Developer tab, AutoCorrect, outline view,
protection, compare, print, accessibility, references and Language/
Hyphenation dialogs, Draw additions). pl.tsv lacked them, so
polish_covers_every_key_of_every_other_catalog and
every_tab_and_command_is_translated failed. All are now translated.

cargo xtask ci: all 6 steps pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Brandon Thomas <echelon@gmail.com>
This commit is contained in:
witkowski01Claude Opus 5.5Brandon Thomas
 authored and GitHub committed 2026-10-12 04:04:03 -04:00
1 parent 55f7b894c6
commit 1fa5333b26
51 files changed
+9837 -191
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

ATTRIBUTION.md
+2
View File
@@ -11,6 +11,7 @@ Generated-in-code art is original and has no file to list: the UI icon set (`cra
| `assets/hyphenation/en-us.pat` | DesignCraft contributors (generated) | trained from the Moby list by DesignCraft's patgen | Public Domain (CC0-1.0) | Our own Liang hyphenation patterns (no TeX patterns) |
| `assets/hyphenation/en-us.dic` | Grady Ward (Moby Hyphenator II word list); converted by DesignCraft contributors | https://www.gutenberg.org/ebooks/3204 (file `mhyph.txt`, SHA-256 `eeb30474c86b8af3469035ec1a0913e35905325ca885290db9dfed9881e230ac`), via DesignCraft | Public Domain | ~165k words with break points; drives spelling and hyphenation |
| `assets/spelling/en-extra.txt` | WordCraft contributors | original word list written for this repository (common modern words the Moby list lacks); no other dictionary copied | MIT OR Apache-2.0 | ~220 lowercase English base forms supplementing spelling (#520) |
| `assets/hyphenation/pl.pat` | Hanna Kołodziejska, Bogusław Jackowski, Marek Ryćko (Polish TeX hyphenation patterns, 1987–1995); hyph-utf8 package | https://github.com/hyphenation/tex-hyphen (`hyph-utf8/tex/generic/hyph-utf8/patterns/tex/hyph-pl.tex`, SHA-256 `83ddbbd21bc06a44bc4f4a4c49c38782e809cff5f026b764c044daa29e818843`), bundled unmodified | MIT (the file is offered under MIT or Knuth's public-domain terms; WordCraft uses MIT, whose notice is in the file's header) | 4,053 Liang patterns and 20 exception words; Polish hyphenation |
| `assets/app-icon/wordcraft-1024.png` | WordCraft contributors | original artwork drawn in SVG (`assets/app-icon/wordcraft.svg`), rendered by `packaging/icons.sh` | MIT OR Apache-2.0 | Placeholder app icon until the WordCraft mascot drawing lands |
| `packaging/macos/dmg/background.svg` | @XusBadia | original work for this repository, derived from the WordCraft app icon (`assets/app-icon/wordcraft.svg`, referenced via `<image>`, not copied); text is outlined Inter / JetBrains Mono glyph paths | MIT OR Apache-2.0, like the code ([`LICENSE-MIT`](LICENSE-MIT), [`LICENSE-APACHE`](LICENSE-APACHE)) | WordCraft macOS DMG window background (2026-10-08); used for the Finder window of the macOS DMG, rendered to `background.tiff` by `packaging/macos/dmg/generate.py` |
| `packaging/macos/dmg/background.tiff` | @XusBadia | Rendered from `packaging/macos/dmg/background.svg` by `packaging/macos/dmg/generate.py` | MIT OR Apache-2.0, like the code ([`LICENSE-MIT`](LICENSE-MIT), [`LICENSE-APACHE`](LICENSE-APACHE)) | 1x + 2x HiDPI TIFF for the macOS DMG window; includes the app icon (see its row) and LittleCMS's built-in sRGB profile ("No copyright, use freely") |
@@ -63,6 +64,7 @@ Generated-in-code art is original and has no file to list: the UI icon set (`cra
| craft-fonts (optional build input, not files in this repo): BIZ UDPGothic, BIZ UDMincho, Shippori Mincho, Noto Sans CJK SC, Noto Sans Arabic | see https://github.com/storytold/craft-fonts/blob/main/ATTRIBUTION.md | embedded only when built with `CRAFT_FONTS_DIR` (all official releases) | OFL-1.1 | Japanese and Chinese UI, Japanese document fallback |
| `crates/ui-egui/src/i18n/uk.tsv` | @dmatviichuk with Codex | Original translation from the English interface labels in this repository; no other product localisation copied | MIT OR Apache-2.0 | Ukrainian interface catalog; spelling checked with external VESUM and ULIF dictionaries, neither bundled |
| `crates/ui-egui/src/i18n/et.tsv` | @henno with Codex | Original translation from the English interface labels in this repository; no other product localisation copied | MIT OR Apache-2.0 | Estonian interface catalog |
| `crates/ui-egui/src/i18n/pl.tsv` | Konrad Witkowski with Claude (AI agent) | Original translation from the English interface labels in this repository; no other product localisation copied | MIT OR Apache-2.0 | Polish interface catalog; not yet reviewed by a second native speaker |
| `crates/ui-egui/src/i18n/pt-br.tsv` | @edurbs with ZCode | Original translation from the English interface labels in this repository; no other product localisation copied | MIT OR Apache-2.0 | Brazilian Portuguese interface catalog; spelling checked with the external pt_BR Hunspell dictionary, not bundled |
| `crates/ui-egui/src/i18n/de.tsv` | @arekown with Kimi | Original translation from the English interface labels in this repository; no other product localisation copied | MIT OR Apache-2.0 | German interface catalog |
| `crates/ui-egui/src/i18n/ru.tsv` | Dmitrii Khlapov with Claude | Original translation from the English interface labels in this repository; no other product localisation copied | MIT OR Apache-2.0 | Russian interface catalog; reviewed by a native speaker |
NOTICE
+3
View File
@@ -23,6 +23,9 @@ list (author, source, license) is in ATTRIBUTION.md.
- English word list (assets/hyphenation/en-us.dic) — derived from the Moby Hyphenator II list by
Grady Ward, public domain (Project Gutenberg eBook #3204). The hyphenation patterns
(assets/hyphenation/en-us.pat) were trained from it and are released as CC0-1.0.
- Polish hyphenation patterns (assets/hyphenation/pl.pat) — hyph-pl.tex from the hyph-utf8
package, Copyright (C) 1987-1995 Hanna Kołodziejska, Bogusław Jackowski, Marek Ryćko, bundled
unmodified and used under the MIT License; the copyright and permission notice is in the file.
- All UI icons are drawn in code (crates/ui-egui/src/icons.rs) and are original work, MIT OR Apache-2.0.
Rust dependencies (egui, eframe, vello_cpu, skrifa, harfrust, krilla, quick-xml, zip, image, …)
README.md
+6 -4
View File
@@ -115,9 +115,9 @@ Every screenshot below is WordCraft itself, rendered offscreen by its own UI tes
| **Pages** | Margins, orientation, size, columns, page/column/section breaks, headers and footers (first page, odd/even), page numbers, watermark, page borders, line numbers, vertical alignment, drop caps, automatic hyphenation |
| **Objects** | Pictures (resize, crop, recolour, brightness/contrast, transparency, background removal, picture styles, rotate), shapes, text boxes, floating position with text wrapping (square, top and bottom, behind or in front of text) |
| **References** | Table of contents, footnotes and endnotes, citations and bibliography (APA, MLA, Chicago, IEEE), captions, table of figures, cross-references, index, table of authorities |
| **Review** | Spelling and grammar with suggestions, thesaurus, word count, comments in margin balloons or a pane, track changes, accept/reject, compare documents, restrict editing, accessibility checker, document inspector |
| **Review** | Spelling and grammar with suggestions (English and Polish, by each run's language), thesaurus, word count, comments in margin balloons or a pane, track changes, accept/reject, compare documents, restrict editing, accessibility checker, document inspector |
| **Mailings** | Mail merge from CSV, merge fields, address block, greeting line, rules, preview, finish to a document; envelopes and labels |
| **View** | Print layout, web layout, draft, read mode, focus, zoom, one/multiple pages, page width, Navigation pane, rulers, gridlines, dark mode; interface in English, 简体中文, 繁體中文, 日本語, Português (Brasil), Español, Українська, Српски / Srpski, Eesti, Türkçe, Русский, Italiano or Deutsch (follows the system language by default) |
| **View** | Print layout, web layout, draft, read mode, focus, zoom, one/multiple pages, page width, Navigation pane, rulers, gridlines, dark mode; interface in English, 简体中文, 繁體中文, 日本語, Português (Brasil), Español, Українська, Српски / Srpski, Eesti, Türkçe, Русский, Italiano, Deutsch or Polski (follows the system language by default) |
| **Files** | .docx read/write (opens in Word), Word 97-2003 .doc import, PDF export, .odt, .rtf, .html, .md, .tex, .txt import/export, page images |
The honest picture, area by area, is in [ROADMAP.md](ROADMAP.md), the
@@ -339,8 +339,10 @@ Copyright (c) 2026 ArtCraft Team and the WordCraft contributors. Required notice
Bundled fonts, icons, images and other assets keep their own open licenses; each one is listed
with its author, source and license in [ATTRIBUTION.md](ATTRIBUTION.md).
The spelling dictionary and hyphenation come from Grady Ward's public-domain Moby Hyphenator II
word list. The sample documents and templates are original text written for WordCraft.
The English spelling dictionary and hyphenation come from Grady Ward's public-domain Moby
Hyphenator II word list. Polish spelling reads the Hunspell `pl_PL` dictionary installed on the system (not bundled) and
Polish hyphenation the TeX patterns by Hanna Kołodziejska, Bogusław Jackowski and Marek Ryćko (MIT
option). The sample documents and templates are original text written for WordCraft.
The ArtCraft name, wordmark and logos in [`docs/brand/`](docs/brand/) are trademarks of the
ArtCraft Team and are not covered by this license. They may be used only unmodified, and only as
ROADMAP.md
+8 -5
View File
@@ -2,7 +2,7 @@
**Stage: alpha** · next: beta, ~15 points (60% → 75% ready for real work) and ~150–230 h away
> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-11 · **Change:** trivial (German interface catalog, #574; previously trivial: Turkish interface landed; previously trivial: formatting revisions landed, #41; previously trivial: Draw tab ink, #307, and Draw Table and Eraser, #303, landed: catalog 402/434; previously minor: readiness table with hours per audience; full number back to the additive weighted sum, 60%; mainstream and essentials numbers added; alpha gate checked; previously major: full re-measure against Word for Mac 16.113.4; restructured to the craftrules progress-docs standard) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4
> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-11 · **Change:** minor (Polish interface and Polish proofing; previously trivial: Turkish interface landed; previously trivial: formatting revisions landed, #41; previously trivial: Draw tab ink, #307, and Draw Table and Eraser, #303, landed: catalog 402/434; previously minor: readiness table with hours per audience; full number back to the additive weighted sum, 60%; mainstream and essentials numbers added; alpha gate checked; previously major: full re-measure against Word for Mac 16.113.4; restructured to the craftrules progress-docs standard) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4
WordCraft aims for complete parity with Microsoft Word, then goes further on speed, openness and
agent control. This page is the summary; the assessment is
@@ -58,7 +58,7 @@ the two partial ones (real-world .docx, printing) complete, roughly.
| File formats | 60% | 185–300 | [file-format-parity.md](docs/file-format-parity.md) |
| Layout and pagination fidelity | 55% | 90–145 | [layout-parity.md](docs/layout-parity.md) |
| Hardware | 40% | 20–40 | [hardware-parity.md](docs/hardware-parity.md) |
| Localization | 25% | 80–130 + native review | [localization-parity.md](docs/localization-parity.md) |
| Localization | 27% | 75–120 + native review | [localization-parity.md](docs/localization-parity.md) |
| Performance | 75% | 10–20 | [layout-parity.md](docs/layout-parity.md#performance) |
| Stability | 50% | 25–45 | [gaps.md](docs/gaps.md) (#3) |
| Platforms | 95% (beyond Word on Linux, BSD, web) | 5–10 | this page |
@@ -85,7 +85,7 @@ the two partial ones (real-world .docx, printing) complete, roughly.
| Pictures, shapes, text boxes, WordArt | 60% | 20–30 |
| Backstage, printing, options | 58% | 12–20 |
| Right-to-left and complex scripts | 50% | 15–25 |
| Proofing (spelling, grammar, thesaurus, languages) | 35% | 25–45 |
| Proofing (spelling, grammar, thesaurus, languages) | 40% | 20–40 |
| Accessibility | 35% | 15–25 |
| East Asian typography | 25% | 15–25 |
| Charts, SmartArt, Draw/ink, 3D models, icons | 3% | 70–110 |
@@ -110,8 +110,9 @@ Weights and evidence per area: [docs/target-app-parity.md](docs/target-app-parit
| Korean | ko | none | 0% |
| Vietnamese | vi | none | 0% |
Also shipped: Traditional Chinese (`zh-hant`), Ukrainian (`uk`), Serbian (`sr`, `sr-latn`, #250) and Russian (`ru`), all partial (~95%). Estonian (`et`), Turkish (`tr`) and Italian (`it`, #594) now have complete 1,307-entry interface catalogs with
system-locale selection and a saved language choice (Italian also has 51 strings none of the others had yet); Estonian, Turkish and Italian proofing are not included. Word ships
Also shipped: Traditional Chinese (`zh-hant`), Ukrainian (`uk`), Serbian (`sr`, `sr-latn`, #250), Russian (`ru`) and Polish (`pl`), all partial (~95%). Estonian (`et`), Turkish (`tr`) and Italian (`it`, #594) now have complete 1,307-entry interface catalogs with
system-locale selection and a saved language choice (Italian also has 51 strings none of the others had yet); Estonian, Turkish and Italian proofing are not included.
Proofing: English and Polish (spelling, grammar, hyphenation), chosen per run from its language. Word ships
30 interface languages and proofing for ~50. Detail: [docs/localization-parity.md](docs/localization-parity.md).
## Upcoming
@@ -133,6 +134,7 @@ Ranked, to beta (detail and remaining milestones in [docs/roadmap.md](docs/roadm
| 2026-10-11 | Hide White Space (#645): `view.hideWhiteSpace` (View › Show checkbox, or double-click the gap between pages) collapses margins, headers and footers between pages to a thin line in Print Layout; screen only, page count unchanged |
| 2026-10-11 | Fonts: Palatino Linotype and Book Antiqua fall back to Palatino (macOS) or TeX Gyre Pagella / P052 / URW Palladio (Linux) before generic serif (#670) |
|---|---|
| 2026-10-11 | Polish: interface (`pl`, the thirteenth language, every catalog key) and proofing — spelling against the installed Hunspell `pl_PL` dictionary (read at runtime, not bundled) through a new Hunspell-style affix engine, suggestions tuned for Polish, Polish grammar rules with Polish messages, TeX Polish hyphenation patterns (MIT option); proofing language per run (`w:lang`) for squiggles, F7, suggestions and hyphenation, languages without tools left unmarked; Review › Language menu and the status bar's language; a Polish interface writes new documents in Polish |
| 2026-10-11 | German (`de`) interface (#574): 2,021 translated entries covering the full zh-hans key set, locale and saved-preference tests, bundled-font checks; no German proofing resources |
| 2026-10-11 | `.doc` import keeps pictures and drawings (#551): inline and floating pictures (PNG, JPEG, DIB, EMF, WMF), shapes, lines, groups and text boxes with position and wrap, from [MS-ODRAW] |
| 2026-10-11 | Tables resize with the mouse (#217): drag a column border (the neighbour makes up the difference; Shift moves one column), a row's bottom border, or the handle past the bottom-right corner to scale the table; a dashed guide while dragging, one undo step per drag; `table.resizeDrag` for agents |
@@ -254,6 +256,7 @@ Ranked, to beta (detail and remaining milestones in [docs/roadmap.md](docs/roadm
| 2026-10-10 | trivial | Progress log: Windows OpenGL fallback without a DirectX 12 driver (#316) |
| 2026-10-10 | minor | Password-protected .docx (#281): files encrypted with a password (agile encryption) open and save; File › Info › Encrypt with Password |
| 2026-10-10 | minor | Relanded clipboard pane, table styles editor, Manage Styles, Asian typography (kinsoku), Group/Ungroup and Shape Effects (#264 #263 #270 #271 #267 #280): catalog 390/431 (90.5%) |
| 2026-10-10 | minor | Polish interface and proofing landed: Localization 25% → 27%, Proofing 35% → 40% |
| 2026-10-10 | minor | Merged main: catalog 384/430 (89.3%), 431 commands; Paste Special, Advanced Find, Column Selection, Style Inspector, custom table styles, gridlines, zoom buttons, touchpad scrolling, File Info, Serbian landed; mainstream 54→55% |
| 2026-10-10 | minor | Readiness table with hours per audience; full number restored to the additive weighted sum (~47% → ~60%): method aligned with the standard, no new evidence; beta distance back to ~15 points and ~150–230 h |
| 2026-10-10 | minor | Mainstream practitioner (~54%) and essentials user (~63%) numbers added; full number recomputed 60% → ~47% with the standard's discounts; beta distance now ~28 points and ~450–700 h; stage stays alpha |
assets/hyphenation/pl.pat
+4154
View File
File diff suppressed because it is too large. Load diff
crates/engine/src/cmd/file.rs
+16 -2
View File
@@ -119,7 +119,14 @@ fn new(s: &mut Session, v: &Value) -> CmdResult {
"letter" => crate::sample::letter(),
"resume" => crate::sample::resume(),
"report" => crate::sample::report(),
_ => Document::new(),
_ => {
// A blank document is written in the editing language (the templates' text is English).
let mut d = Document::new();
if let Some(lang) = &s.editing_lang {
d.styles.default_chr.lang = Some(lang.clone());
}
d
}
};
s.set_document(doc);
s.path = None;
@@ -132,12 +139,19 @@ fn open(s: &mut Session, v: &Value) -> CmdResult {
return sel_result(s);
};
let password = p::str(v, "password");
let (doc, encrypted) = if let Some(data) = p::str(v, "data") {
let (mut doc, encrypted) = if let Some(data) = p::str(v, "data") {
let bytes = super::insert::base64_decode(data).ok_or_else(|| CmdError::Params("bad base64".into()))?;
crate::io::open_bytes_with(path, &bytes, password).map_err(CmdError::Failed)?
} else {
crate::io::open_path_with(std::path::Path::new(path), password).map_err(CmdError::Failed)?
};
// Plain text and Markdown carry no language: they are in the editing language.
let ext = std::path::Path::new(path).extension().and_then(|e| e.to_str()).unwrap_or("").to_ascii_lowercase();
if let Some(lang) = &s.editing_lang
&& matches!(ext.as_str(), "txt" | "text" | "md" | "markdown")
{
doc.styles.default_chr.lang = Some(lang.clone());
}
s.set_document(doc);
s.path = Some(path.into());
// Like Word, a document opened with a password is saved with it again.
crates/engine/src/cmd/format.rs
+3
View File
@@ -602,6 +602,9 @@ pub fn state(s: &Session) -> Value {
"highlight": same(&|r| json!(r.highlight.map(|c| c.hex()))),
"style": style,
"styleName": style_name,
// The proofing language (a BCP 47 tag; null when mixed or unset) and "Do not check".
"lang": same(&|r| json!(r.lang)),
"noProof": all.iter().all(|r| r.no_proof),
})
}
crates/engine/src/cmd/review.rs
+11 -9
View File
@@ -738,15 +738,13 @@ fn word_count(s: &mut Session, v: &Value) -> CmdResult {
}))
}
/// Issues (spelling + grammar) in one paragraph as positions.
/// Issues (spelling + grammar) in one paragraph as positions, each in its run's language and
/// skipping text left unchecked — the same issues the layout underlines.
fn para_issues(s: &Session, story: StoryRef, path: &wordcraft_doc::Path) -> Vec<(Pos, Pos, wordcraft_proof::Issue)> {
let Some(p) = s.doc.para(story, path) else { return Vec::new() };
let text = wordcraft_layout::para::proof_text(p);
let mut v: Vec<wordcraft_proof::Issue> = wordcraft_proof::check_spelling_with(&text, &s.prefs.spelling());
v.extend(wordcraft_proof::check_grammar(&text));
v.sort_by_key(|i| i.start);
v.into_iter()
.filter(|i| !p.run_ranges().any(|(r, c)| r.start < i.end && i.start < r.end && (c.no_proof == Some(true) || c.link.is_some())))
let runs = wordcraft_layout::para::proof_runs(&s.doc, p, None);
wordcraft_layout::para::paragraph_issues(p, &runs, &s.prefs.spelling(), true)
.into_iter()
.map(|i| (Pos { story, path: path.clone(), off: i.start }, Pos { story, path: path.clone(), off: i.end }, i))
.collect()
}
@@ -758,8 +756,12 @@ fn issue_at(s: &Session, at: &Pos) -> Option<(Pos, Pos, wordcraft_proof::Issue)>
fn issue_json(s: &Session, a: &Pos, b: &Pos, i: &wordcraft_proof::Issue) -> Value {
let word = s.doc.para_at(a).and_then(|p| p.text.get(a.off..b.off)).unwrap_or("").to_string();
let sugg = if i.kind == wordcraft_proof::IssueKind::Spelling { wordcraft_proof::suggest(&word, 6) } else { i.suggestions.clone() };
json!({"start": pos_json(a), "end": pos_json(b), "text": word, "kind": if i.kind == wordcraft_proof::IssueKind::Spelling { "spelling" } else { "grammar" }, "message": i.message, "suggestions": sugg})
let sugg = if i.kind == wordcraft_proof::IssueKind::Spelling { wordcraft_proof::suggest_in(&word, 6, i.lang) } else { i.suggestions.clone() };
json!({
"start": pos_json(a), "end": pos_json(b), "text": word,
"kind": if i.kind == wordcraft_proof::IssueKind::Spelling { "spelling" } else { "grammar" },
"message": i.message, "suggestions": sugg, "lang": i.lang.code(),
})
}
/// F7: select the next issue after the caret and return it with suggestions.
crates/engine/src/cmd/tools.rs
+7 -6
View File
@@ -577,7 +577,7 @@ pub fn specs() -> Vec<CommandSpec> {
})
.params(r#"{"format": "decimal|upperRoman|lowerLetter…", "text"?: "%1.", "start"?: n}"#),
CommandSpec::new("review.language", "Language", "Review › Language", language).params(
r#"{"lang"?: "en-US|en-GB|fr-FR|…", "noProof"?: bool, "detect"?: bool (detect language automatically, a preference), "default"?: bool (make `lang` the document's default language instead)}"#,
r#"{"lang"?: "en-US|en-GB|pl-PL|fr-FR|…" (BCP 47; default en-US), "noProof"?: bool, "detect"?: bool (detect language automatically, a preference), "default"?: bool (also the document's default language)}"#,
),
CommandSpec::new("review.showMarkup", "Show Markup", "Review › Tracking", |s, v| {
s.view.show_markup = p::bool(v, "value").unwrap_or(!s.view.show_markup);
@@ -633,8 +633,10 @@ pub fn specs() -> Vec<CommandSpec> {
/// Nothing at all sets US English, as before.
fn language(s: &mut Session, v: &Value) -> CmdResult {
let lang = match p::str(v, "lang") {
Some(l) if !l.is_empty() && l.len() <= 35 && l.chars().all(|c| c.is_ascii_alphanumeric() || c == '-') => Some(l.to_string()),
Some(l) => return Err(CmdError::Params(format!("`{l}` is not a language tag"))),
Some(t) => Some(
wordcraft_proof::lang::normalize_tag(t)
.ok_or_else(|| CmdError::Params(format!("`lang` must be a language tag such as pl-PL or en-US, not {t:?}")))?,
),
None => None,
};
let no_proof = p::bool(v, "noProof");
@@ -643,15 +645,14 @@ fn language(s: &mut Session, v: &Value) -> CmdResult {
s.prefs.detect_language = d;
}
if p::bool(v, "default") == Some(true) {
let l = lang.ok_or_else(|| CmdError::Params("`lang` is required with `default`".into()))?;
let l = lang.clone().ok_or_else(|| CmdError::Params("`default` needs `lang`".into()))?;
s.doc.styles.default_chr.lang = Some(l.clone());
// A Normal style that names its own language follows the new default.
if let Some(n) = s.doc.styles.styles.iter_mut().find(|x| x.id == "Normal")
&& n.chr.lang.is_some()
{
n.chr.lang = Some(l.clone());
n.chr.lang = Some(l);
}
return Ok(json!({"default": l, "detect": s.prefs.detect_language}));
}
let lang = match (lang, no_proof, detect) {
(None, None, None) => Some("en-US".to_string()),
crates/engine/src/session.rs
+25
View File
@@ -297,6 +297,10 @@ pub struct Session {
pub keymap: crate::KeyMap,
/// Custom ribbon and Quick Access Toolbar (`tools.customizeRibbon`); the front end saves it too.
pub ribbon: crate::RibbonLayout,
/// The language new blank documents (and opened plain text) are written in, a BCP 47 tag;
/// `None` is WordCraft's default, US English. The front end sets it from the interface
/// language with [`Session::set_editing_language`] (a Polish interface writes Polish).
pub editing_lang: Option<String>,
/// Editing inside an equation: which one and the caret in it.
pub math: Option<MathEdit>,
/// Equations are typed in LaTeX rather than the linear format.
@@ -460,6 +464,7 @@ impl Session {
prefs: Prefs::default(),
keymap: crate::KeyMap::default(),
ribbon: crate::RibbonLayout::default(),
editing_lang: None,
read_aloud: Default::default(),
password: None,
column: None,
@@ -611,6 +616,24 @@ impl Session {
self.cache.clear();
}
/// Set the language new documents are written in ([`Session::editing_lang`]; `None` or a
/// malformed tag: US English). A document nobody has touched yet — untitled, unchanged, no
/// undo steps, no text — switches too, so the blank document a session starts with follows
/// the interface language. Nothing else changes and no undo step is added.
pub fn set_editing_language(&mut self, tag: Option<&str>) {
let tag = tag.and_then(wordcraft_proof::lang::normalize_tag);
if self.editing_lang == tag {
return;
}
self.editing_lang = tag;
let untouched = self.path.is_none() && !self.dirty && self.history.is_empty() && self.redo.is_empty();
if untouched && self.doc.plain_text(StoryRef::Body).trim().is_empty() && self.doc.paragraph_count() <= 1 {
self.doc.styles.default_chr.lang = Some(self.editing_lang.clone().unwrap_or_else(|| "en-US".into()));
self.rev = self.rev.wrapping_add(1);
self.relayout();
}
}
/// Snapshot for undo before a change.
pub fn checkpoint(&mut self, label: &str) {
if label == "Typing" && self.typing_open {
@@ -794,6 +817,8 @@ impl Session {
prefs: _,
keymap: _,
ribbon: _,
// A preference for new documents, like `prefs`: not part of this document.
editing_lang: _,
// Equation editing mode, like `view`: not part of the document or its history.
math: _,
math_latex: _,
crates/engine/src/tests.rs
+121
View File
@@ -3693,6 +3693,127 @@ fn note_options_per_section_and_convert_notes() {
assert!(s.run("references.convertNotes", &json!({"to": "sideways"})).is_err());
}
/// A tiny synthetic Polish Hunspell dictionary (written for these tests). Real dictionaries are
/// installed on the system, never bundled; every Polish test installs this same one.
fn polish_test_dictionary() {
let aff = "SET UTF-8\nTRY aioeznrwcyszżtkpdmłjlubąęgh\nMAP 2\nMAP oó\nMAP zż\nREP 2\nREP ż rz\nREP rz ż\n";
let dic = "12\npłynie\nrzeka\na\nto\nwiem\nże\ntak\nw\nogóle\nzażółć\ngęślą\njaźń\n";
wordcraft_proof::pl::install_dictionary(aff, dic).unwrap();
}
/// Review › Language and the proofing commands work in Polish end to end: the language makes
/// the document's words Polish, F7 and the issue list use the Polish dictionary and grammar,
/// suggestions come from it, and a suggestion fixes the text.
#[test]
fn polish_proofing_through_the_review_commands() {
polish_test_dictionary();
let mut s = s();
run(&mut s, "document.setText", json!({"text": "Płynie żeka, a wogóle to wiem że tak."}));
// English by default: the Polish words are English misspellings.
assert!(run(&mut s, "review.issues", json!({})).as_array().unwrap().len() >= 5);
run(&mut s, "select.all", json!({}));
run(&mut s, "review.language", json!({"lang": "pl_PL", "default": true}));
assert_eq!(s.doc.styles.default_chr.lang.as_deref(), Some("pl-PL"), "the tag is normalized");
assert_eq!(run(&mut s, "format.state", json!({}))["lang"], "pl-PL");
let issues = run(&mut s, "review.issues", json!({}));
let arr = issues.as_array().unwrap();
let has = |text: &str, kind: &str, sugg: &str| {
arr.iter()
.any(|i| i["text"] == text && i["kind"] == kind && i["lang"] == "pl" && i["suggestions"].as_array().unwrap().iter().any(|x| x == sugg))
};
assert!(has("żeka", "spelling", "rzeka"), "{issues}");
assert!(has("wogóle", "spelling", "w ogóle"), "{issues}");
assert!(has(" że", "grammar", ", że"), "{issues}");
assert_eq!(arr.len(), 3, "{issues}");
assert!(arr.iter().any(|i| i["message"] == "Brak przecinka przed „że”"), "{issues}");
// F7 from the start selects the first issue; Change applies a suggestion.
run(&mut s, "caret.docStart", json!({}));
let first = run(&mut s, "review.spelling", json!({}));
assert_eq!(first["text"], "żeka");
run(&mut s, "review.applySuggestion", json!({"text": "rzeka"}));
assert!(text(&s).starts_with("Płynie rzeka,"), "{}", text(&s));
// Add to Dictionary works for Polish words too.
run(&mut s, "review.addToDictionary", json!({"word": "wogóle"}));
let left = run(&mut s, "review.issues", json!({}));
assert_eq!(left.as_array().unwrap().len(), 1, "{left}");
}
/// `review.language`: tags are validated and normalized, `noProof` alone keeps the language,
/// no parameters keep the old meaning (US English), and `default` needs a language.
#[test]
fn review_language_validates_tags_and_marks_text_unchecked() {
polish_test_dictionary();
let mut s = s();
run(&mut s, "document.setText", json!({"text": "Hello wrold"}));
run(&mut s, "select.all", json!({}));
assert!(s.run("review.language", &json!({"lang": "not a tag!"})).is_err());
assert!(s.run("review.language", &json!({"lang": ""})).is_err());
assert!(s.run("review.language", &json!({"default": true, "noProof": true})).is_err(), "default needs a language");
assert_eq!(text(&s), "Hello wrold", "a refused command changes nothing");
run(&mut s, "review.language", json!({"noProof": true}));
let st = run(&mut s, "format.state", json!({}));
assert_eq!((st["noProof"].clone(), st["lang"].clone()), (json!(true), json!("en-US")));
assert_eq!(run(&mut s, "review.issues", json!({})), json!([]));
run(&mut s, "review.language", json!({"noProof": false}));
assert_eq!(run(&mut s, "review.issues", json!({})).as_array().unwrap().len(), 1);
run(&mut s, "review.language", json!({"lang": "PL"}));
assert_eq!(run(&mut s, "format.state", json!({}))["lang"], "pl");
run(&mut s, "review.language", json!({}));
assert_eq!(run(&mut s, "format.state", json!({}))["lang"], "en-US");
// A language without proofing tools: no marks.
run(&mut s, "review.language", json!({"lang": "fr-FR"}));
assert_eq!(run(&mut s, "review.issues", json!({})), json!([]));
}
/// The editing language (set by the front end from the interface language): new blank
/// documents and opened plain text are written in it; the untouched first document switches,
/// a document with changes never does, and templates stay English.
#[test]
fn new_documents_follow_the_editing_language() {
polish_test_dictionary();
let mut s = s();
s.set_editing_language(Some("pl-PL"));
assert_eq!(s.doc.styles.default_chr.lang.as_deref(), Some("pl-PL"), "the untouched blank document switches");
assert!(!s.dirty && !s.can_undo(), "no change to save or undo");
run(&mut s, "text.insert", json!({"text": "Zażółć gęślą jaźń."}));
assert_eq!(run(&mut s, "review.issues", json!({})), json!([]));
s.set_editing_language(None);
assert_eq!(s.doc.styles.default_chr.lang.as_deref(), Some("pl-PL"), "a document with changes keeps its language");
s.set_editing_language(Some("pl"));
run(&mut s, "file.new", json!({}));
assert_eq!(s.doc.styles.default_chr.lang.as_deref(), Some("pl"));
run(&mut s, "file.new", json!({"template": "letter"}));
assert_eq!(s.doc.styles.default_chr.lang.as_deref(), Some("en-US"), "templates are English text");
let data = crate::cmd::insert::base64_encode("Płynie rzeka.".as_bytes());
run(&mut s, "file.open", json!({"path": "notatka.txt", "data": data}));
assert_eq!(s.doc.styles.default_chr.lang.as_deref(), Some("pl"));
assert_eq!(run(&mut s, "review.issues", json!({})), json!([]));
// Malformed tags fall back to US English.
s.set_editing_language(Some("not a tag"));
assert_eq!(s.editing_lang, None);
}
/// A Polish document keeps its languages through .docx: the default (`w:docDefaults`) and an
/// English run (`w:lang`) come back, and so does the proofing.
#[test]
fn proofing_languages_survive_a_docx_round_trip() {
polish_test_dictionary();
let mut s = s();
run(&mut s, "document.setText", json!({"text": "Płynie żeka. Hello wrold."}));
run(&mut s, "select.all", json!({}));
run(&mut s, "review.language", json!({"lang": "pl-PL", "default": true}));
run(&mut s, "select.text", json!({"text": "Hello wrold."}));
run(&mut s, "review.language", json!({"lang": "en-GB"}));
let bytes = crate::io::save_bytes("polski.docx", &s.doc).unwrap();
let doc = crate::io::open_bytes("polski.docx", &bytes).unwrap();
assert_eq!(doc.styles.default_chr.lang.as_deref(), Some("pl-PL"));
let mut back = Session::new(doc);
let issues = run(&mut back, "review.issues", json!({}));
let found: Vec<(String, String)> =
issues.as_array().unwrap().iter().map(|i| (i["text"].as_str().unwrap().to_string(), i["lang"].as_str().unwrap().to_string())).collect();
assert_eq!(found, [("żeka".to_string(), "pl".to_string()), ("wrold".to_string(), "en".to_string())], "{issues}");
}
#[test]
fn hide_white_space_toggles_view_state_only() {
let mut s = s();
crates/layout/src/para.rs
+57 -20
View File
@@ -994,8 +994,11 @@ pub fn layout_para(p: &Paragraph, env: &ParaEnv) -> ParaLayout {
let math_props = doc.settings.math.clone().unwrap_or_default();
// The byte ranges of runs left out of the layout, in order, adjacent runs merged.
let mut left: Vec<std::ops::Range<usize>> = Vec::new();
// Each run's proofing language and whether it is checked (spelling, grammar, hyphenation).
let mut runs: Vec<ProofRun> = Vec::new();
for (range, props) in p.run_ranges() {
let rc = resolve(props);
runs.push(ProofRun::of(range.clone(), &rc));
let Some(text) = p.text.get(range.clone()) else { continue };
if left_out(&rc, b.env.show_hidden, b.env.hide_deleted) {
match left.last_mut() {
@@ -1297,7 +1300,7 @@ pub fn layout_para(p: &Paragraph, env: &ParaEnv) -> ParaLayout {
text_len: p.text.len(),
has_page_fields,
notes,
issues: env.proofing.map(|o| proof_issues(p, &o)).unwrap_or_default(),
issues: env.proofing.map(|o| proof_issues(p, &runs, &o)).unwrap_or_default(),
drop_cap,
hyph_after: Vec::new(),
bidi_levels: b.levels,
@@ -1309,7 +1312,7 @@ pub fn layout_para(p: &Paragraph, env: &ParaEnv) -> ParaLayout {
enclosures,
};
let auto = env.doc.settings.auto_hyphenation && !pl.rp.suppress_hyphens;
pl.hyph_after = hyphenation_points(&text, to_para, &pl, auto, env.doc.settings.hyphenate_caps);
pl.hyph_after = hyphenation_points(&text, to_para, &pl, auto, env.doc.settings.hyphenate_caps, &runs);
pl.left_out = left;
for k in pl.hyph_after.clone() {
if let Some(c) = pl.clusters.get_mut(k as usize) {
@@ -2074,10 +2077,11 @@ fn hyphen_glyph(pl: &ParaLayout, style: u16, cache: &mut Vec<(u16, u32, f32)>) -
}
/// Clusters after which a line may end with a hyphen: soft hyphens always, and dictionary or
/// pattern hyphenation points of each word when automatic hyphenation is on. Both are found in
/// `text`, the paragraph as laid out (without what the layout leaves out, [`left_out`]), and
/// `to_para` maps the end of a char there to the paragraph's text.
fn hyphenation_points(text: &str, to_para: impl Fn(usize) -> Option<usize>, pl: &ParaLayout, auto: bool, caps: bool) -> Vec<u32> {
/// pattern hyphenation points of each word, in the language of its run ([`ProofRun`]), when
/// automatic hyphenation is on (words in capitals only with `caps`). Both are found in `text`, the paragraph as laid out (without what
/// the layout leaves out, [`left_out`]), and `to_para` maps the end of a char there to the
/// paragraph's text.
fn hyphenation_points(text: &str, to_para: impl Fn(usize) -> Option<usize>, pl: &ParaLayout, auto: bool, caps: bool, runs: &[ProofRun]) -> Vec<u32> {
let mut bytes: Vec<usize> = text.char_indices().filter(|(_, c)| *c == SOFT_HYPHEN).map(|(i, c)| i + c.len_utf8()).collect();
if auto {
let lim = wordcraft_proof::hyphen::Limits::default();
@@ -2095,8 +2099,11 @@ fn hyphenation_points(text: &str, to_para: impl Fn(usize) -> Option<usize>, pl:
&& !w.chars().any(wordcraft_fonts::is_rtl)
&& (caps || !all_caps(w))
{
// The word's language: its first letter's run, found by its place in `p.text`.
let first = w.chars().next().map_or(1, char::len_utf8);
let lang = to_para(a + first).map(|e| lang_at(runs, e.saturating_sub(first))).unwrap_or_default();
let offs: Vec<usize> = w.char_indices().map(|(o, _)| o).collect();
for pt in wordcraft_proof::hyphen::hyphen_points(w, &lim) {
for pt in wordcraft_proof::hyphen::hyphen_points_in(w, &lim, lang) {
if let Some(o) = offs.get(pt) {
bytes.push(a + o);
}
@@ -2274,24 +2281,54 @@ impl ParaLayout {
}
}
/// Spelling and grammar issues in a paragraph (skipping "do not check" and hidden runs).
fn proof_issues(p: &Paragraph, opts: &crate::ProofOptions) -> Vec<(usize, usize, bool)> {
/// One run's proofing: its bytes in the paragraph, its language (`w:lang`, resolved through the
/// styles: the document default, paragraph and character styles, direct formatting) and whether
/// it is left unchecked ("Do not check spelling or grammar", hidden text, hyperlinks).
#[derive(Clone, Debug, PartialEq)]
pub struct ProofRun {
pub range: std::ops::Range<usize>,
pub lang: wordcraft_proof::ProofLang,
pub skip: bool,
}
impl ProofRun {
fn of(range: std::ops::Range<usize>, rc: &ResolvedChar) -> ProofRun {
ProofRun { range, lang: wordcraft_proof::ProofLang::from_tag(rc.lang.as_deref()), skip: rc.no_proof || rc.hidden || rc.link.is_some() }
}
}
/// The proofing runs of a paragraph, in order. `table` is the cell's table-style character
/// formatting ([`CellText::chr`]) for a paragraph in a table, as layout resolves it.
pub fn proof_runs(doc: &Document, p: &Paragraph, table: Option<&CharProps>) -> Vec<ProofRun> {
let style = p.props.style.as_deref();
p.run_ranges().map(|(r, c)| ProofRun::of(r, &doc.styles.resolve_char_in(style, table, c))).collect()
}
/// The language of the run holding byte `off` (English past the last run).
fn lang_at(runs: &[ProofRun], off: usize) -> wordcraft_proof::ProofLang {
let k = runs.partition_point(|r| r.range.end <= off);
runs.get(k).filter(|r| r.range.start <= off).map(|r| r.lang).unwrap_or_default()
}
/// Spelling and grammar issues in a paragraph, in order: each word in its run's language
/// ([`wordcraft_proof::check_text_with`]: what spelling skips, and grammar only with `grammar`),
/// and none touching a run left unchecked.
pub fn paragraph_issues(p: &Paragraph, runs: &[ProofRun], spelling: &wordcraft_proof::Options, grammar: bool) -> Vec<wordcraft_proof::Issue> {
if p.text.trim().is_empty() || p.text.len() > 100_000 {
return Vec::new();
}
let text = proof_text(p);
let skip = |a: usize, b: usize| {
p.run_ranges().any(|(r, c)| r.start < b && a < r.end && (c.no_proof == Some(true) || c.hidden == Some(true) || c.link.is_some()))
};
let mut v: Vec<(usize, usize, bool)> = wordcraft_proof::check_spelling_with(&text, &opts.spelling)
let spans: Vec<(std::ops::Range<usize>, wordcraft_proof::ProofLang)> = runs.iter().map(|r| (r.range.clone(), r.lang)).collect();
let skip = |a: usize, b: usize| runs.iter().any(|r| r.skip && r.range.start < b && a < r.range.end);
wordcraft_proof::check_text_with(&text, &spans, spelling, grammar).into_iter().filter(|i| !skip(i.start, i.end)).collect()
}
/// Spelling and grammar issues in a paragraph as squiggles: (start, end, is grammar).
fn proof_issues(p: &Paragraph, runs: &[ProofRun], opts: &crate::ProofOptions) -> Vec<(usize, usize, bool)> {
paragraph_issues(p, runs, &opts.spelling, opts.grammar)
.into_iter()
.filter(|i| !skip(i.start, i.end))
.map(|i| (i.start, i.end, false))
.collect();
if opts.grammar {
v.extend(wordcraft_proof::check_grammar(&text).into_iter().filter(|i| !skip(i.start, i.end)).map(|i| (i.start, i.end, true)));
}
v
.map(|i| (i.start, i.end, i.kind == wordcraft_proof::IssueKind::Grammar))
.collect()
}
/// Text for proofing with the same byte offsets: inline objects and tracked deletions become
crates/layout/src/tests.rs
+58
View File
@@ -3831,6 +3831,64 @@ fn endnotes_at_section_end_and_footnotes_below_text() {
assert!(below > text.top + text.height && below < bottom - 100.0, "{below} {bottom} {text:?}");
}
/// Each run is hyphenated and proofed in its own language (`w:lang`, resolved through the
/// styles): Polish patterns and dictionary for Polish runs, English ones for English runs.
#[test]
fn hyphenation_and_proofing_follow_each_runs_language() {
// A tiny synthetic Polish dictionary (real ones are installed on the system, not bundled).
wordcraft_proof::pl::install_dictionary("SET UTF-8\nSFX B Y 1\nSFX B y ego y\n", "2\nkonstytucyjny/B\npłynie\n").unwrap();
let text = "Konstytucyjnego żeka typography beautifull";
let mut d = Document::from_text(text);
d.settings.auto_hyphenation = true;
let eng = text.find("typography").unwrap();
d.format_range(&Pos::body(0, 0), &Pos::body(0, eng), &|c| c.lang = Some("pl-PL".into())).unwrap();
d.format_range(&Pos::body(0, eng), &Pos::body(0, text.len()), &|c| c.lang = Some("en-US".into())).unwrap();
let mut c = LayoutCache::new();
let l = layout(&d, &mut c, &LayoutOptions { proofing: true, ..LayoutOptions::default() });
let para = l.pages[0].items.iter().find_map(|i| if let Placed::Lines { para, .. } = i { Some(para.clone()) } else { None }).expect("a paragraph");
// Byte offsets after which a hyphen may go.
let ends: Vec<usize> = para.hyph_after.iter().filter_map(|&k| para.clusters.get(k as usize).map(|c| c.end)).collect();
for prefix in ["Kon", "Konsty", "Konstytu", "Konstytucyj", "Konstytucyjne"] {
assert!(ends.contains(&prefix.len()), "Polish kon-sty-tu-cyj-ne-go: no break after {prefix:?} in {ends:?}");
}
for prefix in ["ty", "typog", "typogra"] {
assert!(ends.contains(&(eng + prefix.len())), "English ty-pog-ra-phy: no break after {prefix:?} in {ends:?}");
}
// Spelling: the Polish misspelling and the English one; Polish words aren't English errors.
let flagged: Vec<&str> = para.issues.iter().filter(|i| !i.2).map(|i| &text[i.0..i.1]).collect();
assert_eq!(flagged, ["żeka", "beautifull"]);
// The layout and the Review commands report the same issues.
let p = d.para(StoryRef::Body, &Path::top(0)).unwrap();
let issues = para::paragraph_issues(p, &para::proof_runs(&d, p, None), &Default::default(), true);
assert_eq!(issues.iter().map(|i| (i.start, i.end, i.kind == wordcraft_proof::IssueKind::Grammar)).collect::<Vec<_>>(), para.issues);
assert_eq!(issues[0].lang, wordcraft_proof::ProofLang::Pl);
assert_eq!(issues[1].lang, wordcraft_proof::ProofLang::En);
}
/// Text in a language WordCraft can't proof gets no spelling marks, and "Do not check spelling
/// or grammar" text none either; the document default language applies to text without its own.
#[test]
fn unproofed_languages_and_no_proof_runs_are_not_marked() {
// A tiny synthetic Polish dictionary (real ones are installed on the system, not bundled).
wordcraft_proof::pl::install_dictionary("SET UTF-8\nSFX B Y 1\nSFX B y ego y\n", "2\nkonstytucyjny/B\npłynie\n").unwrap();
let opts = LayoutOptions { proofing: true, ..LayoutOptions::default() };
let issues = |d: &Document| -> usize {
let l = layout(d, &mut LayoutCache::new(), &opts);
l.pages[0].items.iter().map(|i| if let Placed::Lines { para, .. } = i { para.issues.len() } else { 0 }).sum()
};
let text = "Nous écrivons beaucoup aujourd'hui.";
let mut d = Document::from_text(text);
assert!(issues(&d) > 0, "French words are English misspellings in an English document");
d.styles.default_chr.lang = Some("fr-FR".into());
assert_eq!(issues(&d), 0, "no French dictionary: no marks");
d.styles.default_chr.lang = Some("pl-PL".into());
let mut pl = Document::from_text("Płynie żeka.");
pl.styles.default_chr.lang = Some("pl-PL".into());
assert_eq!(issues(&pl), 1);
pl.format_range(&Pos::body(0, 0), &Pos::body(0, "Płynie żeka.".len()), &|c| c.no_proof = Some(true)).unwrap();
assert_eq!(issues(&pl), 0);
}
/// #451: page totals in the body show the document's page count (the stale cached result is
/// ignored), on every page, like the same fields in the footer.
#[test]
crates/proof/Cargo.toml
+1 -1
View File
@@ -3,7 +3,7 @@ name = "wordcraft-proof"
version.workspace = true
edition.workspace = true
license.workspace = true
description = "WordCraft proofing: spelling (public-domain word list), suggestions, grammar checks, hyphenation"
description = "WordCraft proofing per language: spelling, suggestions, grammar checks and hyphenation for English (public-domain word list) and Polish (installed Hunspell dictionary, TeX patterns)"
[lints]
workspace = true
crates/proof/src/affix.rs
+845
View File
@@ -0,0 +1,845 @@
//! Hunspell-style affix dictionaries — the subset the SJP.PL Polish dictionary (`pl_PL`) uses.
//!
//! A word is correct when it is a stem, a stem with one suffix, a stem with one prefix, or a stem
//! with a prefix and a suffix that both allow cross products; each rule applies only to stems
//! carrying its flag and ending (suffixes) or starting (prefixes) as its condition says. Highly
//! inflected languages store millions of forms this way in a few hundred thousand stems: the
//! Polish dictionary's ~350k stems and ~7,500 rules accept ~4.5 million forms.
//!
//! The rules and stems come from a Hunspell `.aff`/`.dic` pair ([`AffixDict::from_hunspell`]),
//! read at runtime from the dictionaries installed on the system ([`crate::hunspell`]). They can
//! also be stored in WordCraft's compact format ([`AffixDict::to_bytes`]): the rules as text
//! lines and the stems front-coded, raw-deflated together (the Polish dictionary is under 1 MB).
//!
//! Supported: one-character flags (at most 64 distinct), `PFX`/`SFX` with strip, append and
//! condition (`.`, letters, `[abc]`, `[^abc]`), cross products, and the suggestion tables `TRY`,
//! `MAP` and `REP`. Rejected: compounding, continuation classes, long or numeric flags,
//! `NEEDAFFIX`, `FORBIDDENWORD` and the other directives that change which words are correct.
//!
//! Lookups never panic and never allocate per probe: stems sit in one string, found through an
//! open-addressing hash table.
use std::collections::HashMap;
use std::hash::Hasher;
use crate::patterns::{FxHasher, FxMap};
const MAGIC: &str = "WCAFF1";
/// Longest word (in bytes) looked up; longer input is never a dictionary word.
pub const MAX_WORD_BYTES: usize = 120;
/// Empty slot in the stem hash table.
const EMPTY: u32 = u32::MAX;
/// Directives that change which words are correct in ways this engine doesn't implement.
const UNSUPPORTED: &[&str] = &[
"AF",
"AM",
"CHECKSHARPS",
"CIRCUMFIX",
"COMPLEXPREFIXES",
"COMPOUNDBEGIN",
"COMPOUNDEND",
"COMPOUNDFLAG",
"COMPOUNDMIDDLE",
"COMPOUNDMIN",
"COMPOUNDPERMITFLAG",
"COMPOUNDRULE",
"FORBIDDENWORD",
"FULLSTRIP",
"ICONV",
"IGNORE",
"KEEPCASE",
"LEMMA_PRESENT",
"NEEDAFFIX",
"OCONV",
"ONLYINCOMPOUND",
"PSEUDOROOT",
"SUBSTANDARD",
];
/// One character class of a rule's condition.
#[derive(Clone, Debug, PartialEq, Eq)]
enum Cond {
Any,
Char(char),
Set { negated: bool, chars: Box<[char]> },
}
impl Cond {
fn matches(&self, c: char) -> bool {
match self {
Cond::Any => true,
Cond::Char(x) => *x == c,
Cond::Set { negated, chars } => chars.contains(&c) != *negated,
}
}
}
/// Parse a Hunspell condition (`.`, `[^i]e`, `[km]anie`…).
fn parse_cond(text: &str) -> Result<Box<[Cond]>, String> {
if text == "." {
return Ok(Box::new([]));
}
let mut out = Vec::new();
let mut chars = text.chars();
while let Some(c) = chars.next() {
match c {
'.' => out.push(Cond::Any),
'[' => {
let mut set = Vec::new();
let mut negated = false;
let mut closed = false;
let mut first = true;
for d in chars.by_ref() {
if d == ']' {
closed = true;
break;
}
if first && d == '^' {
negated = true;
} else {
set.push(d);
}
first = false;
}
if !closed {
return Err(format!("unclosed `[` in condition `{text}`"));
}
out.push(Cond::Set { negated, chars: set.into_boxed_slice() });
}
']' => return Err(format!("stray `]` in condition `{text}`")),
c => out.push(Cond::Char(c)),
}
}
Ok(out.into_boxed_slice())
}
/// One prefix or suffix rule.
#[derive(Clone, Debug)]
struct Rule {
/// Bit of the rule's flag in the stems' flag masks.
bit: u64,
flag: char,
cross: bool,
strip: Box<str>,
add: Box<str>,
cond: Box<[Cond]>,
cond_text: Box<str>,
}
impl Rule {
/// Does `root` end (suffix rule) or start (prefix rule) as the condition says?
fn cond_matches(&self, root: &str, suffix: bool) -> bool {
if self.cond.is_empty() {
return true;
}
if suffix {
let mut it = root.chars().rev();
self.cond.iter().rev().all(|c| it.next().is_some_and(|x| c.matches(x)))
} else {
let mut it = root.chars();
self.cond.iter().all(|c| it.next().is_some_and(|x| c.matches(x)))
}
}
}
/// Suffix rules sharing an appended string and a stripped one: one stem lookup serves them all.
#[derive(Clone, Debug, Default)]
struct SfxGroup {
strip: Box<str>,
rules: Vec<u32>,
}
/// A stem list with affix rules (see the module docs).
#[derive(Clone, Debug, Default)]
pub struct AffixDict {
/// All stems, concatenated, sorted by bytes.
words: String,
offs: Vec<u32>,
/// Per stem: index into `sets`.
flags: Vec<u16>,
/// Distinct flag masks.
sets: Vec<u64>,
/// Open-addressing hash table of stem indices (`EMPTY` = free).
table: Vec<u32>,
/// Flag char of each bit.
flag_chars: Vec<char>,
prefixes: Vec<Rule>,
suffixes: Vec<Rule>,
/// Appended string → groups of suffix rules by stripped string.
sfx_index: FxMap<Box<str>, Vec<SfxGroup>>,
/// Longest appended suffix, in chars.
max_sfx_chars: usize,
try_chars: String,
map: Vec<Vec<char>>,
rep: Vec<(String, String)>,
/// Provenance and licence lines, kept through [`AffixDict::to_bytes`].
comments: Vec<String>,
}
fn hash_str(s: &str) -> u64 {
let mut h = FxHasher::default();
h.write(s.as_bytes());
h.finish()
}
/// Letter case of a word.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Casing {
/// No capitals (`kot`).
Lower,
/// First letter capital, no other (`Kot`).
Initial,
/// Every cased letter a capital, at least two (`KOT`).
Upper,
/// Anything else (`McDonald`, `kOT`).
Mixed,
}
/// The casing of `w`.
pub fn casing(w: &str) -> Casing {
let (mut upper, mut lower, mut first_upper) = (0usize, 0usize, false);
for (i, c) in w.chars().enumerate() {
if c.is_uppercase() {
upper += 1;
if i == 0 {
first_upper = true;
}
} else if c.is_lowercase() {
lower += 1;
}
}
if upper == 0 {
Casing::Lower
} else if upper == 1 && first_upper {
Casing::Initial
} else if lower == 0 {
Casing::Upper
} else {
Casing::Mixed
}
}
/// `w` with its first letter a capital.
pub fn capitalize(w: &str) -> String {
let mut c = w.chars();
match c.next() {
Some(f) => f.to_uppercase().chain(c).collect(),
None => String::new(),
}
}
/// Parsed rules and tables, shared by the Hunspell reader and the compact format.
#[derive(Default)]
struct Rules {
prefixes: Vec<(char, bool, String, String, String)>,
suffixes: Vec<(char, bool, String, String, String)>,
try_chars: String,
map: Vec<Vec<char>>,
rep: Vec<(String, String)>,
}
/// `0` means empty in Hunspell's strip and append fields.
fn zero_empty(s: &str) -> String {
if s == "0" { String::new() } else { s.to_string() }
}
/// Read Hunspell `.aff` text (already decoded to UTF-8).
fn parse_aff(aff: &str) -> Result<Rules, String> {
let mut r = Rules::default();
let mut cross: HashMap<(bool, char), bool> = HashMap::new();
let mut map_header = false;
let mut rep_header = false;
for (n, line) in aff.lines().enumerate() {
let f: Vec<&str> = line.split_whitespace().collect();
let Some(&dir) = f.first() else { continue };
if dir.starts_with('#') {
continue;
}
let at = |k: usize| f.get(k).copied().unwrap_or("");
match dir {
"SET" | "LANG" | "NAME" | "VERSION" | "HOME" | "WORDCHARS" | "KEY" | "MAXNGRAMSUGS" | "MAXDIFF" | "ONLYMAXDIFF" | "NOSPLITSUGS"
| "SUGSWITHDOTS" | "NOSUGGEST" | "WARN" | "FORBIDWARN" | "MAXCPDSUGS" => {}
"FLAG" => return Err(format!("line {}: FLAG {} is not supported (one-character flags only)", n + 1, at(1))),
"TRY" => r.try_chars = at(1).to_string(),
"MAP" => {
if !map_header && at(1).parse::<usize>().is_ok() {
map_header = true;
} else if !at(1).contains('(') {
r.map.push(at(1).chars().collect());
}
}
"REP" => {
if !rep_header && f.len() == 2 && at(1).parse::<usize>().is_ok() {
rep_header = true;
} else if f.len() >= 3 {
r.rep.push((at(1).replace('_', " "), at(2).replace('_', " ")));
}
}
"PFX" | "SFX" => {
let suffix = dir == "SFX";
let mut flag_chars = at(1).chars();
let (Some(flag), None) = (flag_chars.next(), flag_chars.next()) else {
return Err(format!("line {}: flag `{}` is not one character", n + 1, at(1)));
};
let is_header = f.len() == 4 && matches!(at(2), "Y" | "N") && at(3).parse::<usize>().is_ok() && !cross.contains_key(&(suffix, flag));
if is_header {
cross.insert((suffix, flag), at(2) == "Y");
continue;
}
if f.len() < 4 {
return Err(format!("line {}: short affix rule", n + 1));
}
let add = at(3);
if add.contains('/') {
return Err(format!("line {}: continuation classes are not supported", n + 1));
}
let cond = if f.len() >= 5 { at(4) } else { "." };
parse_cond(cond).map_err(|e| format!("line {}: {e}", n + 1))?;
let c = cross.get(&(suffix, flag)).copied().unwrap_or(false);
let rule = (flag, c, zero_empty(at(2)), zero_empty(add), cond.to_string());
if suffix {
r.suffixes.push(rule);
} else {
r.prefixes.push(rule);
}
}
d if UNSUPPORTED.contains(&d) => return Err(format!("line {}: {d} is not supported", n + 1)),
_ => {}
}
}
Ok(r)
}
impl AffixDict {
/// Build from Hunspell `.aff` and `.dic` text (decoded to UTF-8; the `.dic` count line and
/// morphological fields are ignored). `comments` (provenance, licence) are kept with the data.
pub fn from_hunspell(aff: &str, dic: &str, comments: &[&str]) -> Result<AffixDict, String> {
let rules = parse_aff(aff)?;
let mut stems: Vec<(String, String)> = Vec::new();
for (n, line) in dic.lines().enumerate() {
let Some(tok) = line.split_whitespace().next() else { continue };
if n == 0 && tok.parse::<usize>().is_ok() {
continue;
}
// `\/` is a literal slash in a word; the first plain `/` starts the flags.
let mut word = String::new();
let mut flags = String::new();
let mut chars = tok.chars().peekable();
let mut in_flags = false;
while let Some(c) = chars.next() {
if in_flags {
flags.push(c);
} else if c == '\\' && chars.peek() == Some(&'/') {
word.push('/');
chars.next();
} else if c == '/' {
in_flags = true;
} else {
word.push(c);
}
}
if word.is_empty() || word.len() > MAX_WORD_BYTES || word.contains(['\n', '/']) {
continue;
}
stems.push((word, flags));
}
AffixDict::build(rules, stems, comments.iter().map(|s| s.to_string()).collect())
}
fn build(rules: Rules, mut stems: Vec<(String, String)>, comments: Vec<String>) -> Result<AffixDict, String> {
// Flags: every one a rule or a stem uses gets a bit.
let mut flag_chars: Vec<char> = rules.prefixes.iter().chain(&rules.suffixes).map(|r| r.0).collect();
flag_chars.extend(stems.iter().flat_map(|s| s.1.chars()));
flag_chars.sort_unstable();
flag_chars.dedup();
if flag_chars.len() > 64 {
return Err(format!("{} distinct flags; at most 64 are supported", flag_chars.len()));
}
let bit_of = |c: char| flag_chars.iter().position(|&x| x == c).map_or(0, |i| 1u64 << i);
// Stems sorted by bytes; homonyms (the same word twice) are merged.
stems.sort_by(|a, b| a.0.as_bytes().cmp(b.0.as_bytes()));
let mut d = AffixDict {
words: String::with_capacity(stems.iter().map(|s| s.0.len()).sum()),
offs: Vec::with_capacity(stems.len() + 1),
flag_chars: flag_chars.clone(),
try_chars: rules.try_chars,
map: rules.map,
rep: rules.rep,
comments,
..AffixDict::default()
};
d.offs.push(0);
let mut set_ids: HashMap<u64, u16> = HashMap::new();
let mut i = 0;
while i < stems.len() {
let Some((word, _)) = stems.get(i) else { break };
let mut mask = 0u64;
let mut j = i;
while let Some((w, f)) = stems.get(j)
&& w == word
{
mask |= f.chars().map(bit_of).fold(0, |a, b| a | b);
j += 1;
}
let next_id = u16::try_from(d.sets.len()).map_err(|_| "too many distinct flag sets".to_string())?;
let id = *set_ids.entry(mask).or_insert(next_id);
if usize::from(id) == d.sets.len() {
d.sets.push(mask);
}
d.words.push_str(word);
d.offs.push(u32::try_from(d.words.len()).map_err(|_| "stems too large".to_string())?);
d.flags.push(id);
i = j;
}
let mk = |(flag, cross, strip, add, cond): (char, bool, String, String, String)| -> Result<Rule, String> {
Ok(Rule { bit: bit_of(flag), flag, cross, strip: strip.into(), add: add.into(), cond: parse_cond(&cond)?, cond_text: cond.into() })
};
d.prefixes = rules.prefixes.into_iter().map(mk).collect::<Result<_, _>>()?;
d.suffixes = rules.suffixes.into_iter().map(mk).collect::<Result<_, _>>()?;
d.index();
Ok(d)
}
/// Build the suffix index and the stem hash table.
fn index(&mut self) {
let mut idx: FxMap<Box<str>, Vec<SfxGroup>> = FxMap::default();
for (i, r) in self.suffixes.iter().enumerate() {
let groups = idx.entry(r.add.clone()).or_default();
let Ok(i) = u32::try_from(i) else { break };
match groups.iter_mut().find(|g| g.strip == r.strip) {
Some(g) => g.rules.push(i),
None => groups.push(SfxGroup { strip: r.strip.clone(), rules: vec![i] }),
}
}
self.max_sfx_chars = self.suffixes.iter().map(|r| r.add.chars().count()).max().unwrap_or(0);
self.sfx_index = idx;
let n = self.len();
let size = (n.max(1) * 2).next_power_of_two();
let mut table = vec![EMPTY; size];
let mask = size - 1;
for i in 0..n {
let Some(w) = self.word(i) else { continue };
let mut slot = hash_str(w) as usize & mask;
while table.get(slot).is_some_and(|&t| t != EMPTY) {
slot = (slot + 1) & mask;
}
if let (Some(t), Ok(i)) = (table.get_mut(slot), u32::try_from(i)) {
*t = i;
}
}
self.table = table;
}
/// Number of stems.
pub fn len(&self) -> usize {
self.flags.len()
}
pub fn is_empty(&self) -> bool {
self.flags.is_empty()
}
/// Number of prefix and suffix rules.
pub fn rule_count(&self) -> usize {
self.prefixes.len() + self.suffixes.len()
}
fn word(&self, i: usize) -> Option<&str> {
let a = *self.offs.get(i)? as usize;
let b = *self.offs.get(i + 1)? as usize;
self.words.get(a..b)
}
/// The stems, in byte order.
pub fn stems(&self) -> impl Iterator<Item = &str> + '_ {
(0..self.len()).filter_map(|i| self.word(i))
}
/// The stems starting with `prefix`, in byte order (a binary search, then a scan).
pub fn stems_with_prefix<'a>(&'a self, prefix: &str) -> impl Iterator<Item = &'a str> + use<'a> {
let prefix = prefix.to_string();
let first = {
let (mut lo, mut hi) = (0usize, self.len());
while lo < hi {
let mid = lo + (hi - lo) / 2;
if self.word(mid).is_some_and(|w| w.as_bytes() < prefix.as_bytes()) {
lo = mid + 1;
} else {
hi = mid;
}
}
lo
};
(first..self.len()).map_while(move |i| self.word(i).filter(|w| w.starts_with(prefix.as_str())))
}
/// Flag mask of a stem, if `w` is one.
fn stem_flags(&self, w: &str) -> Option<u64> {
if self.table.is_empty() {
return None;
}
let mask = self.table.len() - 1;
let mut slot = hash_str(w) as usize & mask;
for _ in 0..self.table.len() {
let t = *self.table.get(slot)?;
if t == EMPTY {
return None;
}
if self.word(t as usize) == Some(w) {
let id = *self.flags.get(t as usize)?;
return self.sets.get(usize::from(id)).copied();
}
slot = (slot + 1) & mask;
}
None
}
/// Is `w` exactly (case and all) a stem or an affixed form of one?
pub fn contains(&self, w: &str) -> bool {
if w.is_empty() || w.len() > MAX_WORD_BYTES {
return false;
}
if self.stem_flags(w).is_some() || self.suffixed(w, 0) {
return true;
}
let mut root = String::with_capacity(w.len() + 8);
for r in &self.prefixes {
let Some(rest) = w.strip_prefix(&*r.add) else { continue };
if rest.is_empty() {
continue;
}
root.clear();
root.push_str(&r.strip);
root.push_str(rest);
if !r.cond_matches(&root, false) {
continue;
}
if self.stem_flags(&root).is_some_and(|f| f & r.bit != 0) {
return true;
}
if r.cross && self.suffixed(&root, r.bit) {
return true;
}
}
false
}
/// Is `w` a stem with one suffix? With `need` set (a prefix was taken off), the stem must
/// carry those flags too and the suffix must allow cross products.
fn suffixed(&self, w: &str, need: u64) -> bool {
let mut root = String::with_capacity(w.len() + 8);
// Each split point from the end: the appended string is `w[b..]`, at most the longest
// suffix, and something must stay before it.
let cuts = std::iter::once(w.len()).chain(w.char_indices().rev().map(|(i, _)| i).filter(|&i| i > 0));
for b in cuts.take(self.max_sfx_chars + 1) {
let (Some(base), Some(add)) = (w.get(..b), w.get(b..)) else { continue };
let Some(groups) = self.sfx_index.get(add) else { continue };
for g in groups {
root.clear();
root.push_str(base);
root.push_str(&g.strip);
let Some(f) = self.stem_flags(&root) else { continue };
if f & need != need {
continue;
}
let ok = g
.rules
.iter()
.filter_map(|&i| self.suffixes.get(i as usize))
.any(|r| f & r.bit != 0 && (need == 0 || r.cross) && r.cond_matches(&root, true));
if ok {
return true;
}
}
}
false
}
/// Is `w` correct, Hunspell style: as written, or — for a capitalized word or one in capitals
/// — in lower case (and capitalized), since a sentence may start with any word and headings
/// may be in capitals. Lower-case input never matches a capitalized stem (`warszawa`).
pub fn check(&self, w: &str) -> bool {
if self.contains(w) {
return true;
}
match casing(w) {
Casing::Lower | Casing::Mixed => false,
Casing::Initial => self.contains(&w.to_lowercase()),
Casing::Upper => {
let lower = w.to_lowercase();
self.contains(&capitalize(&lower)) || self.contains(&lower)
}
}
}
/// Every form of the stem `stem` (itself included), unsorted, possibly with repeats.
pub fn forms(&self, stem: &str) -> Vec<String> {
let Some(f) = self.stem_flags(stem) else { return Vec::new() };
let mut out = vec![stem.to_string()];
let mut cross_sfx = Vec::new();
for r in &self.suffixes {
if f & r.bit == 0 || !r.cond_matches(stem, true) {
continue;
}
let Some(base) = stem.strip_suffix(&*r.strip) else { continue };
let form = format!("{base}{}", r.add);
if r.cross {
cross_sfx.push(form.clone());
}
out.push(form);
}
for r in &self.prefixes {
if f & r.bit == 0 || !r.cond_matches(stem, false) {
continue;
}
let Some(rest) = stem.strip_prefix(&*r.strip) else { continue };
out.push(format!("{}{rest}", r.add));
if r.cross {
for s in &cross_sfx {
if let Some(rest) = s.strip_prefix(&*r.strip) {
out.push(format!("{}{rest}", r.add));
}
}
}
}
out
}
/// The `TRY` letters, most frequent first (suggestions try these).
pub fn try_chars(&self) -> &str {
&self.try_chars
}
/// The `MAP` groups: letters often confused with each other (`aą`, `zżź`, `oóu`…).
pub fn map(&self) -> &[Vec<char>] {
&self.map
}
/// The `REP` table: common misspellings, `(wrong, right)`.
pub fn rep(&self) -> &[(String, String)] {
&self.rep
}
/// The provenance and licence lines.
pub fn comments(&self) -> &[String] {
&self.comments
}
/// Serialize to WordCraft's compact format: raw-deflate of `WCAFF1`, the comments, the
/// tables and rules as text lines, a `WORDS` line, then one line per stem, front-coded
/// against the previous one (a byte with the shared prefix length in bytes, the rest of the
/// word, `/` and its flags if any).
pub fn to_bytes(&self) -> Vec<u8> {
let mut out = String::new();
out.push_str(MAGIC);
out.push('\n');
for c in &self.comments {
out.push_str("# ");
out.push_str(c.trim());
out.push('\n');
}
if !self.try_chars.is_empty() {
out.push_str(&format!("TRY {}\n", self.try_chars));
}
for m in &self.map {
out.push_str(&format!("MAP {}\n", m.iter().collect::<String>()));
}
for (a, b) in &self.rep {
out.push_str(&format!("REP {} {}\n", a.replace(' ', "_"), b.replace(' ', "_")));
}
let z = |s: &str| if s.is_empty() { "0".to_string() } else { s.to_string() };
for (kind, rules) in [("PFX", &self.prefixes), ("SFX", &self.suffixes)] {
for r in rules {
let y = if r.cross { "Y" } else { "N" };
out.push_str(&format!("{kind} {} {y} {} {} {}\n", r.flag, z(&r.strip), z(&r.add), r.cond_text));
}
}
out.push_str("WORDS\n");
let mut raw = out.into_bytes();
let mut prev: &[u8] = &[];
for i in 0..self.len() {
let Some(w) = self.word(i) else { continue };
let b = w.as_bytes();
let p = prev.iter().zip(b).take_while(|(x, y)| x == y).count().min(255);
raw.push(p as u8);
raw.extend_from_slice(b.get(p..).unwrap_or_default());
let mask = self.flags.get(i).and_then(|&id| self.sets.get(usize::from(id))).copied().unwrap_or(0);
if mask != 0 {
raw.push(b'/');
for (bit, c) in self.flag_chars.iter().enumerate() {
if mask & (1 << bit) != 0 {
let mut buf = [0u8; 4];
raw.extend_from_slice(c.encode_utf8(&mut buf).as_bytes());
}
}
}
raw.push(b'\n');
prev = b;
}
miniz_oxide::deflate::compress_to_vec(&raw, 10)
}
/// Read data written by [`AffixDict::to_bytes`].
pub fn from_bytes(data: &[u8]) -> Result<AffixDict, String> {
let raw =
miniz_oxide::inflate::decompress_to_vec_with_limit(data, 64 << 20).map_err(|e| format!("affix dictionary: inflate failed: {e:?}"))?;
let words_at = raw.windows(7).position(|w| w == b"\nWORDS\n").ok_or("affix dictionary: no WORDS section")?;
let head = std::str::from_utf8(raw.get(..words_at).unwrap_or_default()).map_err(|_| "affix dictionary: bad utf-8 in the rules")?;
let mut lines = head.lines();
if lines.next() != Some(MAGIC) {
return Err("affix dictionary: bad header".into());
}
let mut rules = Rules::default();
let mut comments = Vec::new();
for line in lines {
if let Some(c) = line.strip_prefix("# ") {
comments.push(c.to_string());
continue;
}
let f: Vec<&str> = line.split_whitespace().collect();
let at = |k: usize| f.get(k).copied().unwrap_or("");
match at(0) {
"TRY" => rules.try_chars = at(1).to_string(),
"MAP" => rules.map.push(at(1).chars().collect()),
"REP" => rules.rep.push((at(1).replace('_', " "), at(2).replace('_', " "))),
kind @ ("PFX" | "SFX") => {
let mut fc = at(1).chars();
let (Some(flag), None) = (fc.next(), fc.next()) else { return Err(format!("affix dictionary: bad flag in `{line}`")) };
if f.len() != 6 {
return Err(format!("affix dictionary: bad rule `{line}`"));
}
let rule = (flag, at(2) == "Y", zero_empty(at(3)), zero_empty(at(4)), at(5).to_string());
if kind == "SFX" {
rules.suffixes.push(rule);
} else {
rules.prefixes.push(rule);
}
}
"" => {}
other => return Err(format!("affix dictionary: unknown line `{other}`")),
}
}
let body = raw.get(words_at + 7..).unwrap_or_default();
let mut stems: Vec<(String, String)> = Vec::new();
let mut cur: Vec<u8> = Vec::new();
let mut i = 0;
while let Some(&p) = body.get(i) {
let rest = body.get(i + 1..).unwrap_or_default();
let len = rest.iter().position(|&b| b == b'\n').ok_or("affix dictionary: truncated")?;
let line = rest.get(..len).unwrap_or_default();
i += len + 2;
let p = p as usize;
if p > cur.len() {
return Err("affix dictionary: bad prefix".into());
}
let (tail, flags) = match line.iter().position(|&b| b == b'/') {
Some(k) => (line.get(..k).unwrap_or_default(), line.get(k + 1..).unwrap_or_default()),
None => (line, &[][..]),
};
cur.truncate(p);
cur.extend_from_slice(tail);
let word = std::str::from_utf8(&cur).map_err(|_| "affix dictionary: bad utf-8 in a stem")?;
let flags = std::str::from_utf8(flags).map_err(|_| "affix dictionary: bad utf-8 in flags")?;
if word.is_empty() || word.len() > MAX_WORD_BYTES {
return Err("affix dictionary: bad stem".into());
}
stems.push((word.to_string(), flags.to_string()));
}
AffixDict::build(rules, stems, comments)
}
}
#[cfg(test)]
mod tests {
use super::*;
const AFF: &str = "SET UTF-8\nTRY aeoiułkr\n\nPFX b Y 1\nPFX b 0 nie .\n\nSFX a Y 3\nSFX a 0 y [^y]\nSFX a ek ka ek\nSFX a 0 em .\n\nSFX c N 1\nSFX c a ą a\n\nMAP 2\nMAP aą\nMAP oóu\nREP 2\nREP ż rz\nREP u ó\n";
const DIC: &str = "4\nkot/ab\npiesek/a\nkrowa/c\nWarszawa/c\n";
fn dict() -> AffixDict {
AffixDict::from_hunspell(AFF, DIC, &["test data"]).unwrap()
}
#[test]
fn stems_suffixes_prefixes_and_cross_products() {
let d = dict();
assert_eq!(d.len(), 4);
for w in ["kot", "koty", "kotem", "piesek", "pieska", "piesekem", "krowa", "krową", "niekot", "niekoty", "niekotem", "Warszawa", "Warszawą"]
{
assert!(d.contains(w), "{w}");
}
for w in ["kotą", "pieseka", "pieskem", "niepiesek", "niekrową", "krowy", "warszawa", "", "nie", "y", "ka"] {
assert!(!d.contains(w), "{w}");
}
}
#[test]
fn case_variants() {
let d = dict();
assert!(d.check("Kot") && d.check("KOT") && d.check("Koty") && d.check("WARSZAWĄ") && d.check("Warszawą"));
assert!(!d.check("warszawa") && !d.check("kOT") && !d.check("KoT"));
assert_eq!(casing("Kot"), Casing::Initial);
assert_eq!(casing("KOT"), Casing::Upper);
assert_eq!(casing("K"), Casing::Initial);
assert_eq!(casing("kot"), Casing::Lower);
assert_eq!(casing("McDonald"), Casing::Mixed);
assert_eq!(capitalize("żółw"), "Żółw");
}
#[test]
fn forms_are_all_accepted() {
let d = dict();
let mut forms = d.forms("kot");
forms.sort();
forms.dedup();
assert_eq!(forms, ["kot", "kotem", "koty", "niekot", "niekotem", "niekoty"]);
for stem in d.stems().map(str::to_string).collect::<Vec<_>>() {
for f in d.forms(&stem) {
assert!(d.contains(&f), "{stem} → {f}");
}
}
}
#[test]
fn compact_format_round_trips() {
let d = dict();
let back = AffixDict::from_bytes(&d.to_bytes()).unwrap();
assert_eq!(back.len(), d.len());
assert_eq!(back.rule_count(), d.rule_count());
assert_eq!(back.comments(), ["test data"]);
assert_eq!(back.rep(), d.rep());
assert_eq!(back.map(), d.map());
assert_eq!(back.try_chars(), "aeoiułkr");
for w in ["kotem", "niekoty", "pieska", "krową", "Warszawą"] {
assert!(back.contains(w), "{w}");
}
assert!(!back.contains("kotą"));
}
#[test]
fn unsupported_and_broken_input_is_an_error() {
assert!(AffixDict::from_hunspell("FLAG long\n", DIC, &[]).is_err());
assert!(AffixDict::from_hunspell("COMPOUNDFLAG X\n", DIC, &[]).is_err());
assert!(AffixDict::from_hunspell("SFX a Y 1\nSFX a 0 y/b .\n", DIC, &[]).is_err());
assert!(AffixDict::from_hunspell("SFX a Y 1\nSFX a 0 y [ab\n", DIC, &[]).is_err());
assert!(AffixDict::from_bytes(b"not deflate").is_err());
let ok = dict().to_bytes();
assert!(AffixDict::from_bytes(&ok[..ok.len() / 2]).is_err());
let garbage = miniz_oxide::deflate::compress_to_vec(b"WCAFF1\nWORDS\n\x09abc\n", 6);
assert!(AffixDict::from_bytes(&garbage).is_err(), "prefix longer than the previous word");
let empty = AffixDict::default();
assert!(!empty.contains("kot") && !empty.check("Kot") && empty.forms("kot").is_empty());
}
proptest::proptest! {
#[test]
fn lookups_never_panic(s in "\\PC{0,40}", bytes in proptest::collection::vec(proptest::num::u8::ANY, 0..200)) {
let d = dict();
let _ = d.check(&s);
let _ = d.forms(&s);
let _ = AffixDict::from_bytes(&bytes);
let _ = AffixDict::from_bytes(&miniz_oxide::deflate::compress_to_vec(&bytes, 1));
}
}
}
crates/proof/src/hunspell.rs
+163
View File
@@ -0,0 +1,163 @@
//! Hunspell dictionaries installed on the system, read at runtime.
//!
//! WordCraft doesn't bundle large spelling dictionaries. For languages with an affix dictionary
//! (Polish), it reads the standard Hunspell `.aff`/`.dic` pair installed by the operating system,
//! a package manager or LibreOffice, with [`crate::affix::AffixDict::from_hunspell`]. When none is
//! installed the language has no spelling check (no marks, no errors), like Word without the
//! language's proofing tools.
//!
//! Search order: the directory in `WORDCRAFT_DICTIONARIES` (or one set with
//! [`set_dictionary_dir`]), then the usual system locations. On the web there are none.
use std::path::{Path, PathBuf};
use std::sync::RwLock;
use crate::affix::AffixDict;
/// Largest `.aff` or `.dic` file read (real ones are a few MB).
const MAX_FILE: u64 = 64 << 20;
/// An extra directory searched first (a setting), set by the front end.
static EXTRA_DIR: RwLock<Option<PathBuf>> = RwLock::new(None);
/// Search `dir` before the system locations (a "dictionaries folder" setting); `None` clears it.
/// Takes effect for dictionaries not loaded yet.
pub fn set_dictionary_dir(dir: Option<PathBuf>) {
*EXTRA_DIR.write().unwrap_or_else(|e| e.into_inner()) = dir;
}
/// Directories that may hold Hunspell dictionaries, most specific first.
pub fn search_dirs() -> Vec<PathBuf> {
let mut dirs: Vec<PathBuf> = Vec::new();
if let Some(d) = EXTRA_DIR.read().unwrap_or_else(|e| e.into_inner()).clone() {
dirs.push(d);
}
if cfg!(target_arch = "wasm32") {
return dirs;
}
if let Some(d) = std::env::var_os("WORDCRAFT_DICTIONARIES").filter(|v| !v.is_empty()) {
dirs.push(PathBuf::from(d));
}
let home = std::env::var_os("HOME").filter(|v| !v.is_empty()).map(PathBuf::from);
if cfg!(windows) {
for var in ["LOCALAPPDATA", "APPDATA"] {
if let Some(base) = std::env::var_os(var).filter(|v| !v.is_empty()).map(PathBuf::from) {
dirs.push(base.join("WordCraft").join("dictionaries"));
dirs.push(base.join("hunspell"));
}
}
for var in ["ProgramFiles", "ProgramFiles(x86)"] {
if let Some(base) = std::env::var_os(var).filter(|v| !v.is_empty()).map(PathBuf::from) {
dirs.push(base.join("LibreOffice").join("share").join("extensions").join("dict-pl"));
}
}
} else if cfg!(target_os = "macos") {
if let Some(h) = &home {
dirs.push(h.join("Library/Spelling"));
}
dirs.push(PathBuf::from("/Library/Spelling"));
dirs.push(PathBuf::from("/opt/homebrew/share/hunspell"));
dirs.push(PathBuf::from("/usr/local/share/hunspell"));
dirs.push(PathBuf::from("/Applications/LibreOffice.app/Contents/Resources/extensions/dict-pl"));
} else {
if let Some(h) = &home {
dirs.push(h.join(".local/share/hunspell"));
}
for d in [
"/usr/share/hunspell",
"/usr/share/myspell",
"/usr/share/myspell/dicts",
"/usr/local/share/hunspell",
"/usr/local/share/myspell",
"/usr/lib/libreoffice/share/extensions/dict-pl",
"/opt/libreoffice/share/extensions/dict-pl",
] {
dirs.push(PathBuf::from(d));
}
}
dirs
}
/// The first `<name>.aff`/`<name>.dic` pair found for any of `names` (`pl_PL`, `pl`), searching
/// [`search_dirs`].
pub fn find(names: &[&str]) -> Option<(PathBuf, PathBuf)> {
search_dirs().into_iter().find_map(|dir| find_in(&dir, names))
}
/// The first `<name>.aff`/`<name>.dic` pair for any of `names` in `dir`.
pub fn find_in(dir: &Path, names: &[&str]) -> Option<(PathBuf, PathBuf)> {
names.iter().find_map(|n| {
let aff = dir.join(format!("{n}.aff"));
let dic = dir.join(format!("{n}.dic"));
(aff.is_file() && dic.is_file()).then_some((aff, dic))
})
}
/// Read a Hunspell pair from disk (size-capped), decoded per the `.aff`'s `SET` line.
pub fn load(aff: &Path, dic: &Path) -> Result<AffixDict, String> {
let read = |p: &Path| -> Result<Vec<u8>, String> {
let len = std::fs::metadata(p).map_err(|e| format!("{}: {e}", p.display()))?.len();
if len > MAX_FILE {
return Err(format!("{}: too large ({len} bytes)", p.display()));
}
std::fs::read(p).map_err(|e| format!("{}: {e}", p.display()))
};
let (aff_text, dic_text) = decode(&read(aff)?, &read(dic)?)?;
let source = format!("Hunspell dictionary {}", dic.display());
AffixDict::from_hunspell(&aff_text, &dic_text, &[source.as_str()])
}
/// ISO 8859-2 code points 0xA0..=0xFF (0x00..=0x9F map to themselves).
const ISO_8859_2_HIGH: &str = "\u{a0}Ą˘Ł¤ĽŚ§¨ŠŞŤŹ\u{ad}ŽŻ°ą˛ł´ľśˇ¸šşťź˝žżŔÁÂĂÄĹĆÇČÉĘËĚÍÎĎĐŃŇÓÔŐÖ×ŘŮÚŰÜÝŢßŕáâăäĺćçčéęëěíîďđńňóôőö÷řůúűüýţ˙";
fn decode_latin2(bytes: &[u8]) -> String {
let high: Vec<char> = ISO_8859_2_HIGH.chars().collect();
bytes.iter().map(|&b| if b < 0xa0 { char::from(b) } else { high.get(usize::from(b - 0xa0)).copied().unwrap_or('\u{fffd}') }).collect()
}
/// Decode an `.aff`/`.dic` pair to UTF-8 from the encoding the `.aff`'s `SET` line names
/// (UTF-8 when there is none; ISO 8859-2 is the other one Polish dictionaries use).
pub fn decode(aff: &[u8], dic: &[u8]) -> Result<(String, String), String> {
let head = String::from_utf8_lossy(aff.get(..aff.len().min(8192)).unwrap_or_default()).to_string();
let set = head.lines().find_map(|l| l.trim_start_matches('\u{feff}').strip_prefix("SET ")).map(str::trim).unwrap_or("UTF-8").to_ascii_uppercase();
match set.as_str() {
"UTF-8" | "UTF8" => {
let text = |b: &[u8], what: &str| String::from_utf8(b.to_vec()).map_err(|e| format!("{what}: not UTF-8: {e}"));
Ok((text(aff, ".aff")?, text(dic, ".dic")?))
}
"ISO8859-2" | "ISO-8859-2" => Ok((decode_latin2(aff), decode_latin2(dic))),
other => Err(format!("unsupported encoding {other}")),
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn decodes_latin2_and_utf8() {
let aff = b"SET ISO8859-2\nTRY a\n";
let (a, d) = decode(aff, b"2\n\xbf\xf3\xb3w\n").unwrap();
assert!(a.contains("TRY a"));
assert_eq!(d, "2\nżółw\n");
let (_, d) = decode("SET UTF-8\n".as_bytes(), "żółw".as_bytes()).unwrap();
assert_eq!(d, "żółw");
assert!(decode(b"SET KOI8-R\n", b"").is_err());
assert!(decode(b"SET UTF-8\n", b"\xff\xfe").is_err());
}
#[test]
fn finds_and_loads_a_pair_in_a_directory() {
let dir = std::env::temp_dir().join(format!("wordcraft-hunspell-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
assert!(find_in(&dir, &["xx_XX"]).is_none(), "no files: nothing found");
std::fs::write(dir.join("xx_XX.aff"), "SET UTF-8\nSFX A Y 1\nSFX A 0 y .\n").unwrap();
assert!(find_in(&dir, &["xx_XX"]).is_none(), "an .aff alone isn't a dictionary");
std::fs::write(dir.join("xx_XX.dic"), "1\nkot/A\n").unwrap();
let (aff, dic) = find_in(&dir, &["yy", "xx_XX"]).unwrap();
let d = load(&aff, &dic).unwrap();
assert!(d.check("kot") && d.check("koty") && !d.check("kotx"));
assert!(load(&dir.join("missing.aff"), &dic).is_err());
let _ = std::fs::remove_dir_all(&dir);
}
}
crates/proof/src/hyphen.rs
+183 -12
View File
@@ -1,6 +1,7 @@
//! English hyphenation.
//! Hyphenation, per proofing language ([`ProofLang`]).
//!
//! Lookup order for each word:
//! **English** (and languages without patterns of their own, as before languages were told
//! apart), lookup order for each word:
//! 1. the **dictionary** ([`Dictionary::en_us`]) — the public-domain Moby Hyphenator II word list
//! by Grady Ward (~160k words), stored compactly in `assets/hyphenation/en-us.dic`; simple
//! inflections (`'s`, `-s`, `-es`, `-ed`, `-d`) inherit the breaks of their stem;
@@ -8,6 +9,10 @@
//! set, trained from the same list by DesignCraft's patgen (generated by DesignCraft's `hyphgen` example);
//! 3. a small syllable **heuristic** for words outside the pattern alphabet.
//!
//! **Polish**: the exception words, then the TeX Polish patterns ([`Patterns::pl`], by Hanna
//! Kołodziejska, Bogusław Jackowski and Marek Ryćko, MIT licence option), then a Polish syllable
//! heuristic for words with letters outside the Polish alphabet (`é`, `ü`…).
//!
//! Then the paragraph's limits ([`Limits`]) are applied.
pub use crate::dict;
@@ -16,11 +21,14 @@ pub use crate::patterns;
use std::sync::OnceLock;
pub use crate::lang::ProofLang;
pub use dict::Dictionary;
pub use patterns::Patterns;
static EN_US_DIC: &[u8] = include_bytes!("../../../assets/hyphenation/en-us.dic");
static EN_US_PAT: &str = include_str!("../../../assets/hyphenation/en-us.pat");
/// hyph-utf8's `hyph-pl.tex`, unmodified (its licence header is in the file).
static PL_PAT: &str = include_str!("../../../assets/hyphenation/pl.pat");
impl Dictionary {
/// The bundled US English dictionary (decoded on first use).
@@ -44,6 +52,12 @@ impl Patterns {
static P: OnceLock<Patterns> = OnceLock::new();
P.get_or_init(|| Patterns::parse(EN_US_PAT))
}
/// The bundled Polish patterns and exception words (parsed on first use).
pub fn pl() -> &'static Patterns {
static P: OnceLock<Patterns> = OnceLock::new();
P.get_or_init(|| Patterns::parse(PL_PAT))
}
}
/// Hyphenation limits from the paragraph's Hyphenation settings.
@@ -79,11 +93,14 @@ fn is_word_char(c: char) -> bool {
}
/// Unrestricted break points of one lowercase core (letters and `'` only, no leading/trailing `'`).
fn core_breaks(core: &[char]) -> (Vec<usize>, Source) {
fn core_breaks(core: &[char], lang: ProofLang) -> (Vec<usize>, Source) {
let n = core.len();
if n < 2 {
return (vec![], Source::None);
}
if lang == ProofLang::Pl {
return core_breaks_pl(core);
}
let dict = Dictionary::en_us();
let s: String = core.iter().collect();
let from_mask = |m: u64, len: usize| (1..len.min(64)).filter(|&i| m & (1 << i) != 0).collect::<Vec<_>>();
@@ -108,10 +125,30 @@ fn core_breaks(core: &[char]) -> (Vec<usize>, Source) {
(heuristic_points(core), Source::Heuristic)
}
/// Unrestricted break points of `word` (char indices; a hyphen goes before that char), handling
/// case, apostrophes (`don't`, `’s`), surrounding punctuation and compounds (each letter run is
/// hyphenated separately). No break is ever placed next to an apostrophe or non-letter.
/// Polish: exception words, then the patterns, then the Polish syllable heuristic.
fn core_breaks_pl(core: &[char]) -> (Vec<usize>, Source) {
let pats = Patterns::pl();
let s: String = core.iter().collect();
if let Some(p) = pats.exception(&s) {
return (p.to_vec(), Source::Dictionary);
}
if !pats.is_empty()
&& let Some(p) = pats.points(core)
{
return (p, Source::Patterns);
}
(heuristic_points_pl(core), Source::Heuristic)
}
/// Unrestricted break points of an English `word` ([`word_breaks_in`] with [`ProofLang::En`]).
pub fn word_breaks(word: &str) -> (Vec<usize>, Source) {
word_breaks_in(word, ProofLang::En)
}
/// Unrestricted break points of `word` in `lang` (char indices; a hyphen goes before that char),
/// handling case, apostrophes (`don't`, `’s`), surrounding punctuation and compounds (each letter
/// run is hyphenated separately). No break is ever placed next to an apostrophe or non-letter.
pub fn word_breaks_in(word: &str, lang: ProofLang) -> (Vec<usize>, Source) {
let chars: Vec<char> = word.chars().collect();
let mut out = Vec::new();
let mut src = Source::None;
@@ -133,7 +170,7 @@ pub fn word_breaks(word: &str) -> (Vec<usize>, Source) {
b -= 1;
}
let core: Vec<char> = chars[a..b].iter().map(|&c| if is_apostrophe(c) { '\'' } else { c.to_lowercase().next().unwrap_or(c) }).collect();
let (pts, s) = core_breaks(&core);
let (pts, s) = core_breaks(&core, lang);
if src == Source::None {
src = s;
}
@@ -143,11 +180,16 @@ pub fn word_breaks(word: &str) -> (Vec<usize>, Source) {
(out, src)
}
/// Allowed hyphenation points in `word` as char indices (a hyphen goes before that char), with
/// the paragraph limits applied to each letter run.
/// Allowed hyphenation points in an English `word` ([`hyphen_points_in`] with [`ProofLang::En`]).
pub fn hyphen_points(word: &str, lim: &Limits) -> Vec<usize> {
hyphen_points_in(word, lim, ProofLang::En)
}
/// Allowed hyphenation points in `word`, a word in `lang`, as char indices (a hyphen goes before
/// that char), with the paragraph limits applied to each letter run.
pub fn hyphen_points_in(word: &str, lim: &Limits, lang: ProofLang) -> Vec<usize> {
let chars: Vec<char> = word.chars().collect();
let (pts, _) = word_breaks(word);
let (pts, _) = word_breaks_in(word, lang);
if pts.is_empty() {
return pts;
}
@@ -208,7 +250,12 @@ pub fn hyphen_points(word: &str, lim: &Limits) -> Vec<usize> {
/// `word` with `-` at every hyphenation point (tests and diagnostics).
pub fn hyphenate_word(word: &str, lim: &Limits) -> String {
let pts = hyphen_points(word, lim);
hyphenate_word_in(word, lim, ProofLang::En)
}
/// `word`, a word in `lang`, with `-` at every hyphenation point (tests and diagnostics).
pub fn hyphenate_word_in(word: &str, lim: &Limits, lang: ProofLang) -> String {
let pts = hyphen_points_in(word, lim, lang);
let mut s = String::with_capacity(word.len() + pts.len());
for (i, ch) in word.chars().enumerate() {
if pts.contains(&i) {
@@ -221,8 +268,14 @@ pub fn hyphenate_word(word: &str, lim: &Limits) -> String {
// ---------- heuristic fallback (words outside the pattern alphabet) ----------
/// Letters with a diacritic that are consonants (`ł`, `ż`, `ç`, `ñ`…); any other non-ASCII letter
/// counts as a vowel (`é`, `ø`, `ą`…), and so do letters of other scripts, which keeps the
/// heuristic from breaking them.
const NON_ASCII_CONSONANTS: &str = "łńśćźżçčďđğģķļľĺňñņřŕšşșťţțžßŋðþ";
fn is_vowel(c: char) -> bool {
matches!(c.to_ascii_lowercase(), 'a' | 'e' | 'i' | 'o' | 'u' | 'y') || (c.is_alphabetic() && !c.is_ascii())
let l = c.to_lowercase().next().unwrap_or(c);
if l.is_ascii() { matches!(l, 'a' | 'e' | 'i' | 'o' | 'u' | 'y') } else { l.is_alphabetic() && !NON_ASCII_CONSONANTS.contains(l) }
}
fn onset_pair(a: char, b: char) -> bool {
@@ -286,6 +339,52 @@ fn heuristic_points(c: &[char]) -> Vec<usize> {
out
}
/// Polish digraphs, one sound each: never split.
fn is_digraph_pl(a: char, b: char) -> bool {
matches!((a, b), ('c', 'h' | 'z') | ('d', 'z' | 'ź' | 'ż') | ('r' | 's', 'z'))
}
/// Polish syllable heuristic, for words with letters outside the Polish pattern alphabet (`é`,
/// `ü`…): one break between each pair of syllable nuclei — after a leading sonorant (`kon-ty`,
/// `Pol-ska`) or inside a doubled consonant (`Pöp-pel`), else before the consonant cluster
/// (`sio-stra`). Digraphs (`ch`, `cz`, `dz`, `dź`, `dż`, `rz`, `sz`) stay whole, and an `i` that
/// only softens the consonant before a vowel (`nia`, `sio`) stays with that consonant.
fn heuristic_points_pl(c: &[char]) -> Vec<usize> {
let n = c.len();
if n < 4 || !c.iter().all(|c| c.is_alphabetic()) {
return vec![];
}
let v: Vec<bool> = c.iter().map(|&x| is_vowel(x)).collect();
let at = |i: usize| v.get(i).copied().unwrap_or(false);
// A syllable nucleus: a vowel, except a softening `i` (after a consonant, before a vowel).
let nucleus: Vec<bool> = (0..n).map(|i| at(i) && !(c.get(i) == Some(&'i') && i > 0 && !at(i - 1) && at(i + 1))).collect();
let nuclei: Vec<usize> = (0..n).filter(|&i| nucleus.get(i).copied().unwrap_or(false)).collect();
let mut out = vec![];
for pair in nuclei.windows(2) {
let [a, b] = pair else { continue };
let (start, end) = (a + 1, *b);
if start >= end {
continue; // adjacent vowels (na-u-ka): no break inside a vowel pair here
}
// Consonant units of the cluster: digraphs and a softening `i` join their consonant.
let mut units: Vec<usize> = Vec::new();
let mut i = start;
while i < end {
units.push(i);
i += if i + 1 < end && c.get(i).zip(c.get(i + 1)).is_some_and(|(x, y)| is_digraph_pl(*x, *y)) { 2 } else { 1 };
if i < end && c.get(i) == Some(&'i') {
i += 1;
}
}
let sonorant = |i: usize| matches!(c.get(i), Some('l' | 'ł' | 'm' | 'n' | 'ń' | 'r' | 'j'));
let first_is_single = units.get(1).is_some_and(|&second| second == start + 1);
// A doubled consonant splits (`lek-ki`, `Pöp-pel`), as does a cluster after a sonorant.
let doubled = c.get(start).is_some_and(|x| c.get(start + 1) == Some(x));
out.push(if units.len() >= 2 && first_is_single && (sonorant(start) || doubled) { start + 1 } else { start });
}
out
}
#[cfg(test)]
mod tests {
use super::*;
@@ -527,4 +626,76 @@ mod tests {
let c: Vec<char> = "happen".chars().collect();
assert_eq!(heuristic_points(&c), vec![3]);
}
#[test]
fn heuristic_knows_consonants_with_diacritics() {
// `ł`, `ż`, `ç` are consonants (every non-ASCII letter used to count as a vowel).
assert!(!is_vowel('ł') && !is_vowel('Ż') && !is_vowel('ç') && is_vowel('é') && is_vowel('ą') && is_vowel('ø'));
let c: Vec<char> = "façade".chars().collect();
assert_eq!(heuristic_points(&c), vec![2]);
}
fn pl(w: &str) -> String {
hyphenate_word_in(w, &Limits::default(), ProofLang::Pl)
}
#[test]
fn polish_patterns_load() {
let p = Patterns::pl();
assert!(p.len() > 4000, "{}", p.len());
assert_eq!(p.exception_count(), 20);
assert_eq!(p.exception("nigdy"), Some(&[2][..]));
}
#[test]
fn polish_words_hyphenate_by_polish_rules() {
for (w, want) in [
("konstytucja", "kon-sty-tu-cja"),
("konstytucyjnego", "kon-sty-tu-cyj-ne-go"),
("Rzeczpospolita", "Rzecz-po-spo-li-ta"),
("przedsiębiorstwo", "przed-się-bior-stwo"),
("pięćdziesięciu", "pięć-dzie-się-ciu"),
("dziewczyna", "dziew-czy-na"),
("książka", "książ-ka"),
("Warszawa", "War-sza-wa"),
("szczęście", "szczę-ście"),
("rzeczywistość", "rze-czy-wi-stość"),
("najpiękniejszy", "naj-pięk-niej-szy"),
("wyobraźnia", "wy-obraź-nia"),
("gżegżółka", "gżeg-żół-ka"),
// One syllable, or a single letter left over: no break.
("chrząszcz", "chrząszcz"),
("źdźbło", "źdźbło"),
("obiad", "obiad"),
// The pattern file's exception words.
("nigdy", "ni-gdy"),
("przynajmniej", "przy-naj-mniej"),
("skądinąd", "skąd-inąd"),
// Compounds: each part, never next to the hyphen.
("biało-czerwony", "bia-ło-czer-wo-ny"),
] {
assert_eq!(pl(w), want, "{w}");
}
assert_eq!(word_breaks_in("nigdy", ProofLang::Pl).1, Source::Dictionary);
assert_eq!(word_breaks_in("konstytucja", ProofLang::Pl).1, Source::Patterns);
}
#[test]
fn polish_heuristic_for_foreign_letters() {
// Letters outside the Polish alphabet: the Polish syllable heuristic, digraphs kept whole.
assert_eq!(word_breaks_in("Pöppelmann", ProofLang::Pl).1, Source::Heuristic);
assert_eq!(pl("Pöppelmann"), "Pöp-pel-mann");
assert_eq!(pl("Müllerowi"), "Mül-le-ro-wi");
let c: Vec<char> = "kawiarnié".chars().collect();
assert_eq!(heuristic_points_pl(&c), vec![2, 6], "ka-wiar-nié: `wi` and `ni` soften, `rn` splits after the sonorant");
let c: Vec<char> = "szczérze".chars().collect();
assert_eq!(heuristic_points_pl(&c), vec![5], "szczé-rze: `rz` stays whole");
}
#[test]
fn languages_without_patterns_keep_the_english_ones() {
for w in ["hyphenation", "typography", "internationalization"] {
assert_eq!(hyphen_points_in(w, &Limits::default(), ProofLang::Other), hyphen_points(w, &Limits::default()), "{w}");
}
}
}
crates/proof/src/lang.rs
+131
View File
@@ -0,0 +1,131 @@
//! Proofing languages: which dictionary, grammar rules and hyphenation patterns apply to a run of
//! text. A run's language is its BCP 47 tag (`w:lang` in .docx, `CharProps::lang` in the model),
//! resolved with [`ProofLang::from_tag`].
/// A language WordCraft can proof, or [`ProofLang::Other`] for one it can't.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash, PartialOrd, Ord)]
pub enum ProofLang {
/// English, any region (`en`, `en-US`, `en-GB`…): the public-domain Moby word list, the
/// English grammar rules and the English hyphenation dictionary and patterns. Also text with
/// no language at all (WordCraft's default editing language is US English, like Word's).
#[default]
En,
/// Polish (`pl`, `pl-PL`): the installed Hunspell dictionary (when there is one), the Polish grammar rules and the Polish
/// hyphenation patterns.
Pl,
/// A language without proofing tools here (`fr-FR`, `ja-JP`, `x-none`…). Like Word without
/// the language's proofing tools: no spelling or grammar marks, so its words are never
/// flagged as English misspellings. Hyphenation keeps using the English patterns, as before
/// languages were told apart.
Other,
}
impl ProofLang {
/// Every language with a spelling dictionary, in display order.
pub const PROOFED: [ProofLang; 2] = [ProofLang::En, ProofLang::Pl];
/// The proofing language of a BCP 47 tag (`pl-PL`, `en_GB`, `PL`…). Only the primary subtag
/// counts. No tag (or an empty one) is English, WordCraft's default editing language.
pub fn from_tag(tag: Option<&str>) -> ProofLang {
let Some(tag) = tag.map(str::trim).filter(|t| !t.is_empty()) else { return ProofLang::En };
let primary = tag.split(['-', '_']).next().unwrap_or("");
if primary.eq_ignore_ascii_case("en") || primary.eq_ignore_ascii_case("eng") {
ProofLang::En
} else if primary.eq_ignore_ascii_case("pl") || primary.eq_ignore_ascii_case("pol") {
ProofLang::Pl
} else {
ProofLang::Other
}
}
/// The primary language subtag (`en`, `pl`); empty for [`ProofLang::Other`].
pub fn code(self) -> &'static str {
match self {
ProofLang::En => "en",
ProofLang::Pl => "pl",
ProofLang::Other => "",
}
}
/// Does WordCraft check spelling and grammar in this language?
pub fn is_proofed(self) -> bool {
self != ProofLang::Other
}
}
/// A BCP 47 tag in canonical case (`pl-PL`, `en-GB`, `zh-Hant-TW`, `x-none`), or `None` when
/// `tag` isn't shaped like one: 1–35 ASCII letters, digits and `-`/`_` separators, starting with
/// a 2–8 letter language (or `x`/`i` for private and grandfathered tags). `_` becomes `-`.
pub fn normalize_tag(tag: &str) -> Option<String> {
let tag = tag.trim();
if tag.is_empty() || tag.len() > 35 || !tag.chars().all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_') {
return None;
}
let parts: Vec<&str> = tag.split(['-', '_']).collect();
if parts.iter().any(|p| p.is_empty() || p.len() > 8) {
return None;
}
let first = parts.first()?;
let language = (2..=8).contains(&first.len()) && first.chars().all(|c| c.is_ascii_alphabetic());
if !language && !first.eq_ignore_ascii_case("x") && !first.eq_ignore_ascii_case("i") {
return None;
}
let letters = |p: &str| p.chars().all(|c| c.is_ascii_alphabetic());
// After a singleton (`x-…` private use, `u-…` extensions) subtags are lower case.
let singleton = parts.iter().position(|p| p.len() == 1).unwrap_or(usize::MAX);
let out: Vec<String> = parts
.iter()
.enumerate()
.map(|(i, p)| {
if i >= singleton {
p.to_ascii_lowercase()
} else if i > 0 && p.len() == 2 && letters(p) {
p.to_ascii_uppercase() // region
} else if i > 0 && p.len() == 4 && letters(p) {
let lower = p.to_ascii_lowercase();
let mut c = lower.chars();
c.next().map(|f| f.to_ascii_uppercase().to_string() + c.as_str()).unwrap_or_default() // script
} else {
p.to_ascii_lowercase()
}
})
.collect();
Some(out.join("-"))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn tags_normalize() {
assert_eq!(normalize_tag("PL-pl").as_deref(), Some("pl-PL"));
assert_eq!(normalize_tag(" pl_pl ").as_deref(), Some("pl-PL"));
assert_eq!(normalize_tag("pl").as_deref(), Some("pl"));
assert_eq!(normalize_tag("zh-hant-tw").as_deref(), Some("zh-Hant-TW"));
assert_eq!(normalize_tag("es-419").as_deref(), Some("es-419"));
assert_eq!(normalize_tag("x-none").as_deref(), Some("x-none"));
assert_eq!(normalize_tag("de-DE-u-co-PHONEBK").as_deref(), Some("de-DE-u-co-phonebk"));
for bad in ["", " ", "p", "1pl", "pl--PL", "pl PL", "pl-PL;", "polski język", "-pl", "pl-", "abcdefghi", &"a".repeat(40)] {
assert_eq!(normalize_tag(bad), None, "{bad:?}");
}
}
#[test]
fn tags_resolve() {
for t in ["pl", "pl-PL", "PL-pl", "pl_PL", "pl_PL.UTF-8", " pl-PL ", "pol"] {
assert_eq!(ProofLang::from_tag(Some(t)), ProofLang::Pl, "{t}");
}
for t in ["en", "en-US", "en-GB", "EN_au", "en-Latn-US"] {
assert_eq!(ProofLang::from_tag(Some(t)), ProofLang::En, "{t}");
}
for t in ["fr-FR", "de", "x-none", "ja-JP", "plx", "e", "-"] {
assert_eq!(ProofLang::from_tag(Some(t)), ProofLang::Other, "{t}");
}
assert_eq!(ProofLang::from_tag(None), ProofLang::En);
assert_eq!(ProofLang::from_tag(Some("")), ProofLang::En);
assert_eq!(ProofLang::from_tag(Some(" ")), ProofLang::En);
assert!(ProofLang::Pl.is_proofed() && !ProofLang::Other.is_proofed());
assert_eq!(ProofLang::Pl.code(), "pl");
}
}
crates/proof/src/lib.rs
+241 -35
View File
@@ -1,20 +1,34 @@
//! WordCraft proofing.
//! WordCraft proofing, per language. A run's language (its BCP 47 tag) picks the tools through
//! [`ProofLang`]; the `*_in` functions take it, the older English-only ones remain.
//!
//! - **Spelling** against the public-domain Moby word list (~160k words, the same list that
//! drives hyphenation) plus our supplement of modern words it lacks
//! (`assets/spelling/en-extra.txt`: `desktop`, `email`, `download`…), with inflections (`-s`,
//! `-es`, `-ed`, `-ing`, `-ly`, `'s`…), a user dictionary and ignore lists; suggestions by edit
//! distance.
//! - **Grammar** checks that are cheap and reliable: repeated words, `a`/`an`, capital letter
//! after a sentence end, spaces before punctuation, doubled spaces.
//! - **Hyphenation** ([`hyphen`]): dictionary, Liang patterns, heuristic.
//! - **English** ([`ProofLang::En`]; also text with no language): spelling against the
//! public-domain Moby word list (~160k words, the same list that drives hyphenation) plus our
//! supplement of modern words it lacks (`assets/spelling/en-extra.txt`: `desktop`, `email`,
//! `download`…), with inflections (`-s`, `-es`, `-ed`, `-ing`, `-ly`, `'s`…), a user dictionary and ignore lists;
//! suggestions by edit distance. Grammar checks that are cheap and reliable: repeated words,
//! `a`/`an`, capital letter after a sentence end, spaces before punctuation, doubled spaces.
//! - **Polish** ([`ProofLang::Pl`], module [`pl`]): spelling against the Hunspell `pl_PL`
//! dictionary installed on the system ([`hunspell`]; not bundled, so without it Polish has no
//! spelling marks), read by the affix engine ([`affix`]), suggestions tuned for Polish
//! (diacritics, `ż`/`rz`, `u`/`ó`, missing spaces), and Polish grammar rules with Polish
//! messages.
//! - **Other languages** ([`ProofLang::Other`]): no spelling or grammar marks, like Word without
//! the language's proofing tools.
//! - **Hyphenation** ([`hyphen`]): dictionary, Liang patterns (English, Polish), heuristic.
//!
//! [`check_text`] checks a paragraph whose runs are in different languages.
#![deny(clippy::unwrap_used, clippy::expect_used, clippy::panic, clippy::unimplemented, clippy::todo, clippy::unreachable)]
pub mod affix;
pub mod dict;
pub mod hunspell;
pub mod hyphen;
pub mod lang;
pub mod patterns;
pub mod pl;
use std::collections::HashSet;
use std::ops::Range;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{OnceLock, RwLock};
@@ -35,14 +49,20 @@ impl Default for Options {
}
}
pub use lang::ProofLang;
/// A problem found in text: byte range, kind, message and suggestions.
#[derive(Clone, Debug, PartialEq)]
pub struct Issue {
pub start: usize,
pub end: usize,
pub kind: IssueKind,
/// In the language of the text it was found in (Polish rules explain in Polish).
pub message: String,
pub suggestions: Vec<String>,
/// The language the text was checked in: spelling suggestions come from its dictionary
/// ([`suggest_in`]).
pub lang: ProofLang,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
@@ -59,7 +79,7 @@ fn user_words() -> &'static RwLock<HashSet<String>> {
/// Bumped whenever the user dictionary changes, so cached proofing results can be dropped.
static DICTIONARY_GENERATION: AtomicU64 = AtomicU64::new(0);
/// Add a word to the user dictionary ("Add to Dictionary").
/// Add a word to the user dictionary ("Add to Dictionary"). It applies to every language.
pub fn add_word(w: &str) {
if user_words().write().unwrap_or_else(|e| e.into_inner()).insert(w.to_lowercase()) {
DICTIONARY_GENERATION.fetch_add(1, Ordering::Relaxed);
@@ -80,6 +100,11 @@ pub fn dictionary_generation() -> u64 {
DICTIONARY_GENERATION.load(Ordering::Relaxed)
}
/// Is `lower` (a lower-case word) in the user dictionary?
pub(crate) fn in_user_dictionary(lower: &str) -> bool {
user_words().read().unwrap_or_else(|e| e.into_inner()).contains(lower)
}
/// Words in the user dictionary.
pub fn user_dictionary() -> Vec<String> {
let mut v: Vec<String> = user_words().read().unwrap_or_else(|e| e.into_inner()).iter().cloned().collect();
@@ -161,24 +186,28 @@ fn known_core(w: &str) -> bool {
false
}
/// Is `word` spelled correctly? Numbers, single letters, words in capitals and words with
/// digits are accepted (Word's default [`Options`]).
/// Is `word` spelled correctly? Numbers, single letters, ALL-CAPS acronyms, URLs and words with
/// digits are accepted.
pub fn is_correct(word: &str) -> bool {
is_correct_with(word, &Options::default())
}
/// The words every language's check accepts under `opts`: single letters, words without
/// letters, words with digits (Ignore words that contain numbers) and words in capitals
/// (Ignore words in UPPERCASE).
fn accepted_by_options(word: &str, opts: &Options) -> bool {
let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\'').trim_matches('\'');
w.chars().count() <= 1
|| !w.chars().any(char::is_alphabetic)
|| (opts.ignore_numbers && w.chars().any(|c| c.is_ascii_digit()))
|| (opts.ignore_uppercase && w.chars().all(|c| !c.is_lowercase()))
}
/// [`is_correct`] with the File › Options › Proofing choices.
pub fn is_correct_with(word: &str, opts: &Options) -> bool {
let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\'');
let w = w.trim_matches('\'');
if w.chars().count() <= 1 || !w.chars().any(char::is_alphabetic) {
return true;
}
let digits = w.chars().any(|c| c.is_ascii_digit());
if digits && opts.ignore_numbers {
return true;
}
if opts.ignore_uppercase && w.chars().all(|c| !c.is_lowercase()) {
if accepted_by_options(w, opts) {
return true;
}
if !w.is_ascii() && !w.chars().any(|c| c.is_ascii_alphabetic()) {
@@ -299,31 +328,129 @@ pub fn words(text: &str) -> Vec<(usize, usize)> {
.collect()
}
/// Spelling issues in a paragraph of text (no suggestions; ask [`suggest`] on demand).
/// Spelling issues in a paragraph of English text (no suggestions; ask [`suggest`] on demand).
pub fn check_spelling(text: &str) -> Vec<Issue> {
check_spelling_with(text, &Options::default())
}
/// [`check_spelling`] with the File › Options › Proofing choices.
pub fn check_spelling_with(text: &str, opts: &Options) -> Vec<Issue> {
let mut v = Vec::new();
for (a, b) in words(text) {
let Some(w) = text.get(a..b) else { continue };
// Skip URLs/emails.
let ts = text.get(..a).and_then(|t| t.rfind(char::is_whitespace)).map(|i| i + 1).unwrap_or(0);
let te = text.get(b..).and_then(|t| t.find(char::is_whitespace)).map(|i| b + i).unwrap_or(text.len());
let token = text.get(ts..te).unwrap_or("");
if opts.ignore_internet && (token.contains("://") || token.contains('@') || token.starts_with("www.")) {
continue;
}
if !is_correct_with(w, opts) {
v.push(Issue { start: a, end: b, kind: IssueKind::Spelling, message: "Possible spelling mistake".into(), suggestions: Vec::new() });
words(text).into_iter().filter_map(|(a, b)| spelling_issue(text, a, b, ProofLang::En, opts)).collect()
}
/// Spelling issues in a paragraph of text in `lang` (none for a language without a dictionary;
/// no suggestions: ask [`suggest_in`] on demand).
pub fn check_spelling_in(text: &str, lang: ProofLang) -> Vec<Issue> {
if !lang.is_proofed() {
return Vec::new();
}
words(text).into_iter().filter_map(|(a, b)| spelling_issue(text, a, b, lang, &Options::default())).collect()
}
/// The spelling issue of the word at `a..b` of `text`, if it is misspelled in `lang`. Words in
/// web and e-mail addresses are skipped (unless `opts` says otherwise).
fn spelling_issue(text: &str, a: usize, b: usize, lang: ProofLang, opts: &Options) -> Option<Issue> {
let w = text.get(a..b)?;
let ts = text.get(..a).and_then(|t| t.rfind(char::is_whitespace)).map(|i| i + 1).unwrap_or(0);
let te = text.get(b..).and_then(|t| t.find(char::is_whitespace)).map(|i| b + i).unwrap_or(text.len());
let token = text.get(ts..te).unwrap_or("");
let internet = token.contains("://") || token.contains('@') || token.starts_with("www.");
if (opts.ignore_internet && internet) || is_correct_in_with(w, lang, opts) {
return None;
}
let message = match lang {
ProofLang::Pl => "Możliwy błąd pisowni",
_ => "Possible spelling mistake",
};
Some(Issue { start: a, end: b, kind: IssueKind::Spelling, message: message.into(), suggestions: Vec::new(), lang })
}
/// Is `word` spelled correctly in `lang`? Always true in a language without a dictionary.
pub fn is_correct_in(word: &str, lang: ProofLang) -> bool {
is_correct_in_with(word, lang, &Options::default())
}
/// [`is_correct_in`] with the File › Options › Proofing choices.
pub fn is_correct_in_with(word: &str, lang: ProofLang, opts: &Options) -> bool {
match lang {
ProofLang::En => is_correct_with(word, opts),
ProofLang::Pl => accepted_by_options(word, opts) || pl::is_correct(word),
ProofLang::Other => true,
}
}
/// Spelling suggestions for a word misspelled in `lang` (best first, at most `max`).
pub fn suggest_in(word: &str, max: usize, lang: ProofLang) -> Vec<String> {
match lang {
ProofLang::En => suggest(word, max),
ProofLang::Pl => pl::suggest(word, max),
ProofLang::Other => Vec::new(),
}
}
/// Grammar issues in a paragraph of text in `lang`: the English rules, the Polish ones
/// ([`pl::grammar`]), or none for a language WordCraft doesn't proof.
pub fn check_grammar_in(text: &str, lang: ProofLang) -> Vec<Issue> {
match lang {
ProofLang::En => check_grammar(text),
ProofLang::Pl => pl::grammar::check(text),
ProofLang::Other => Vec::new(),
}
}
/// Spelling and grammar issues in a paragraph whose text is in several languages, in order.
/// `spans` gives the language of byte ranges of `text` (in order, not overlapping); text outside
/// them is English. Each word is spelled in the language of its first letter. Each language's
/// grammar rules read the whole paragraph (so sentence boundaries are seen) but report only
/// inside that language's spans: an English quotation in a Polish paragraph gets English
/// spelling and is never told a Polish rule.
pub fn check_text(text: &str, spans: &[(Range<usize>, ProofLang)]) -> Vec<Issue> {
check_text_with(text, spans, &Options::default(), true)
}
/// [`check_text`] with the File › Options › Proofing choices: what spelling skips, and whether
/// grammar is checked at all.
pub fn check_text_with(text: &str, spans: &[(Range<usize>, ProofLang)], opts: &Options, grammar: bool) -> Vec<Issue> {
// Sorted, empty ones dropped, neighbours of one language merged.
let mut sorted: Vec<&(Range<usize>, ProofLang)> = spans.iter().filter(|(r, _)| r.start < r.end).collect();
sorted.sort_by_key(|s| s.0.start);
let mut merged: Vec<(Range<usize>, ProofLang)> = Vec::new();
for (r, l) in sorted {
match merged.last_mut() {
Some((m, ml)) if *ml == *l && m.end == r.start => m.end = r.end,
_ => merged.push((r.clone(), *l)),
}
}
let lang_at = |i: usize| -> ProofLang {
let k = merged.partition_point(|(r, _)| r.end <= i);
merged.get(k).filter(|(r, _)| r.start <= i).map_or(ProofLang::En, |s| s.1)
};
let mut v: Vec<Issue> = words(text).into_iter().filter_map(|(a, b)| spelling_issue(text, a, b, lang_at(a), opts)).collect();
if !grammar {
v.sort_by_key(|i| (i.start, i.end));
return v;
}
let mut langs: Vec<ProofLang> = merged.iter().map(|s| s.1).collect();
let mut covered = 0;
for (r, _) in &merged {
if r.start > covered {
break;
}
covered = covered.max(r.end);
}
if covered < text.len() || merged.is_empty() {
langs.push(ProofLang::En);
}
langs.sort();
langs.dedup();
for lang in langs {
v.extend(check_grammar_in(text, lang).into_iter().filter(|i| lang_at(i.start) == lang));
}
v.sort_by_key(|i| (i.start, i.end));
v
}
/// Grammar issues in a paragraph of text.
/// Grammar issues in a paragraph of English text.
pub fn check_grammar(text: &str) -> Vec<Issue> {
let mut v = Vec::new();
let ws = words(text);
@@ -341,6 +468,7 @@ pub fn check_grammar(text: &str) -> Vec<Issue> {
kind: IssueKind::Grammar,
message: format!("Repeated word: \"{w1}\""),
suggestions: vec![w0.to_string()],
lang: ProofLang::En,
});
}
if gap == " " && (w0 == "a" || w0 == "A") && w1.chars().next().is_some_and(|c| "aeiouAEIOU".contains(c)) && !starts_consonant_sound(w1) {
@@ -350,6 +478,7 @@ pub fn check_grammar(text: &str) -> Vec<Issue> {
kind: IssueKind::Grammar,
message: "Use \"an\" before a vowel sound".into(),
suggestions: vec![if w0 == "A" { "An" } else { "an" }.into()],
lang: ProofLang::En,
});
}
if gap == " "
@@ -363,6 +492,7 @@ pub fn check_grammar(text: &str) -> Vec<Issue> {
kind: IssueKind::Grammar,
message: "Use \"a\" before a consonant sound".into(),
suggestions: vec![if w0 == "An" { "A" } else { "a" }.into()],
lang: ProofLang::En,
});
}
}
@@ -377,13 +507,21 @@ pub fn check_grammar(text: &str) -> Vec<Issue> {
kind: IssueKind::Grammar,
message: "Remove the space before the punctuation".into(),
suggestions: vec![text.get(i + 1..i + 2).unwrap_or("").to_string()],
lang: ProofLang::En,
});
}
for (i, _) in text.match_indices(" ") {
if i > 0 && text.get(..i).is_some_and(|t| t.ends_with(' ')) {
continue;
}
v.push(Issue { start: i, end: i + 2, kind: IssueKind::Grammar, message: "Extra space".into(), suggestions: vec![" ".into()] });
v.push(Issue {
start: i,
end: i + 2,
kind: IssueKind::Grammar,
message: "Extra space".into(),
suggestions: vec![" ".into()],
lang: ProofLang::En,
});
}
// Sentence start capital.
let mut after_end = true;
@@ -397,6 +535,7 @@ pub fn check_grammar(text: &str) -> Vec<Issue> {
kind: IssueKind::Grammar,
message: "Capitalize the first word of a sentence".into(),
suggestions: vec![fixed],
lang: ProofLang::En,
});
}
let tail = text.get(*b..).unwrap_or("");
@@ -541,6 +680,51 @@ mod tests {
assert!(hyphen::hyphenate_word("hyphenation", &l).matches('-').count() >= 2);
}
fn flagged(t: &str, v: &[Issue]) -> Vec<String> {
v.iter().filter(|i| i.kind == IssueKind::Spelling).map(|i| t[i.start..i.end].to_string()).collect()
}
#[test]
fn each_language_uses_its_own_tools() {
pl::test_dict::install();
let pl = "Zażółć gęślą jaźń, chrząszcz brzmi w trzcinie.";
assert!(check_spelling_in(pl, ProofLang::Pl).is_empty(), "{:?}", check_spelling_in(pl, ProofLang::Pl));
assert!(check_spelling_in(pl, ProofLang::En).len() >= 3, "Polish words are English misspellings");
assert!(check_spelling_in("Nous écrivons en français.", ProofLang::Other).is_empty());
assert!(check_grammar_in("this is is wrong", ProofLang::Other).is_empty());
assert!(is_correct_in("anything", ProofLang::Other) && suggest_in("xyzzy", 5, ProofLang::Other).is_empty());
let v = check_spelling_in("Płynie żeka.", ProofLang::Pl);
assert_eq!(flagged("Płynie żeka.", &v), ["żeka"]);
assert_eq!((v[0].lang, v[0].message.as_str()), (ProofLang::Pl, "Możliwy błąd pisowni"));
assert!(suggest_in("żeka", 5, ProofLang::Pl).contains(&"rzeka".to_string()));
assert!(suggest_in("recieve", 5, ProofLang::En).contains(&"receive".to_string()));
// Polish grammar speaks Polish and has no English a/an rule.
let g = check_grammar_in("Wiem że to jest a apple.", ProofLang::Pl);
assert!(g.iter().any(|i| i.message == "Brak przecinka przed „że”"), "{g:?}");
assert!(!g.iter().any(|i| i.message.contains("\"an\"")), "{g:?}");
}
#[test]
fn mixed_language_paragraphs_check_each_run_in_its_language() {
pl::test_dict::install();
// A Polish paragraph quoting English: English spelling and rules inside the quote only.
let t = "Napisał „the quick brown fox is is here” i poszedł spać, bo żeka wylała.";
let (a, b) = (t.find("the").unwrap(), t.find('”').unwrap());
let spans = [(0..a, ProofLang::Pl), (a..b, ProofLang::En), (b..t.len(), ProofLang::Pl)];
let v = check_text(t, &spans);
assert_eq!(flagged(t, &v), ["żeka"], "{v:?}");
let rep = v.iter().find(|i| i.kind == IssueKind::Grammar).unwrap();
assert_eq!((rep.lang, rep.message.as_str()), (ProofLang::En, "Repeated word: \"is\""), "{v:?}");
// The English rules don't run over the Polish text (no "Capitalize" for the lower-case
// Polish words), and the Polish rules don't run over the English quote.
assert!(v.iter().all(|i| i.lang == ProofLang::Pl || (a..b).contains(&i.start)), "{v:?}");
// Text outside every span is English; no spans at all is English.
assert_eq!(flagged("helo świat", &check_text("helo świat", &[])), ["helo", "świat"]);
assert_eq!(flagged("helo świat", &check_text("helo świat", &[(5..12, ProofLang::Pl)])), ["helo"]);
// A language without tools: nothing flagged there.
assert!(check_text("Bonjour mes amis", &[(0..16, ProofLang::Other)]).is_empty());
}
proptest::proptest! {
#[test]
fn never_panics(s in "\\PC{0,80}") {
@@ -548,6 +732,28 @@ mod tests {
let _ = check_grammar(&s);
let _ = suggest(&s, 3);
let _ = hyphen::hyphen_points(&s, &hyphen::Limits::default());
for lang in [ProofLang::En, ProofLang::Pl, ProofLang::Other] {
let _ = check_spelling_in(&s, lang);
let _ = check_grammar_in(&s, lang);
let _ = is_correct_in(&s, lang);
let _ = hyphen::hyphen_points_in(&s, &hyphen::Limits::default(), lang);
}
let _ = lang::normalize_tag(&s);
let _ = ProofLang::from_tag(Some(&s));
}
#[test]
fn polish_suggestions_never_panic(s in "[a-ząćęłńóśźżA-ZĄĆĘŁŃÓŚŹŻ'’ -]{0,14}") {
let _ = suggest_in(&s, 4, ProofLang::Pl);
}
#[test]
fn mixed_spans_never_panic(s in "\\PC{0,60}", cuts in proptest::collection::vec((0usize..80, 0usize..80, 0u8..3), 0..6)) {
let langs = [ProofLang::En, ProofLang::Pl, ProofLang::Other];
let spans: Vec<(Range<usize>, ProofLang)> = cuts.iter().map(|&(a, b, l)| (a.min(b)..a.max(b), langs[l as usize % 3])).collect();
for i in check_text(&s, &spans) {
proptest::prop_assert!(i.start <= i.end && i.end <= s.len());
}
}
}
}
crates/proof/src/patterns.rs
+139 -33
View File
@@ -1,18 +1,22 @@
//! Liang hyphenation patterns (TeX notation, e.g. `.ab1c`, `4t1ion`): parsing and application.
//!
//! The bundled English set is *ours*: trained from the public-domain Moby Hyphenator word list by
//! DesignCraft's patgen (see `examples/hyphgen.rs`). Patterns are stored in a hash map keyed by the
//! packed letter string, so applying them to a word costs `O(len × max_pattern_len)` lookups.
//! DesignCraft's patgen (see `examples/hyphgen.rs`). The Polish set is the TeX one by Hanna
//! Kołodziejska, Bogusław Jackowski and Marek Ryćko (hyph-utf8 `hyph-pl.tex`, used under its MIT
//! licence option), bundled unmodified: [`Patterns::parse`] reads TeX's `\patterns{…}` and
//! `\hyphenation{…}` blocks directly. Patterns are stored in a hash map keyed by the packed letter
//! string, so applying them to a word costs `O(len × max_pattern_len)` lookups.
use std::collections::HashMap;
use std::hash::{BuildHasherDefault, Hasher};
/// Word-boundary marker in patterns.
pub const DOT: char = '.';
/// Longest pattern the packed key supports.
pub const MAX_PATTERN_LEN: usize = 9;
/// Longest pattern the packed key supports (6 bits a letter in a `u128`; the Polish set's longest
/// is 14 including the dots).
pub const MAX_PATTERN_LEN: usize = 20;
/// Small, fast multiplicative hasher for packed `u64` keys (FxHash-style).
/// Small, fast multiplicative hasher for packed integer keys and short strings (FxHash-style).
#[derive(Default, Clone, Copy)]
pub struct FxHasher(u64);
@@ -21,13 +25,27 @@ impl Hasher for FxHasher {
self.0
}
fn write(&mut self, bytes: &[u8]) {
for &b in bytes {
let (chunks, rest) = bytes.as_chunks::<8>();
for c in chunks {
self.write_u64(u64::from_le_bytes(*c));
}
for &b in rest {
self.write_u64(b as u64);
}
}
fn write_u8(&mut self, i: u8) {
self.write_u64(i as u64);
}
fn write_usize(&mut self, i: usize) {
self.write_u64(i as u64);
}
fn write_u64(&mut self, i: u64) {
self.0 = (self.0.rotate_left(5) ^ i).wrapping_mul(0x51_7c_c1_b7_27_22_0a_95);
}
fn write_u128(&mut self, i: u128) {
self.write_u64(i as u64);
self.write_u64((i >> 64) as u64);
}
}
pub type FxMap<K, V> = HashMap<K, V, BuildHasherDefault<FxHasher>>;
@@ -72,36 +90,67 @@ impl Alphabet {
}
}
/// Packed key of a code string (≤ [`MAX_PATTERN_LEN`] codes).
/// Packed key of a code string (≤ [`MAX_PATTERN_LEN`] codes): 6 bits a code, the length on top.
#[inline]
pub fn pack(codes: &[u8]) -> u64 {
let mut k = 0u64;
pub fn pack(codes: &[u8]) -> u128 {
let mut k = 0u128;
for &c in codes {
k = (k << 6) | c as u64;
k = (k << 6) | c as u128;
}
k | ((codes.len() as u64) << 58)
k | ((codes.len() as u128) << 120)
}
/// A compiled pattern set.
/// A compiled pattern set, with its exception words (TeX's `\hyphenation{…}`).
#[derive(Clone, Debug, Default)]
pub struct Patterns {
alpha: Alphabet,
/// Packed letters → inter-letter values (`len + 1` entries).
map: FxMap<u64, Box<[u8]>>,
map: FxMap<u128, Box<[u8]>>,
max_len: usize,
/// Lowercase word → its break points (char indices), overriding the patterns.
exceptions: HashMap<String, Box<[usize]>>,
}
/// Which TeX block [`Patterns::parse`] is reading.
#[derive(Clone, Copy, PartialEq, Eq)]
enum Block {
Patterns,
Exceptions,
}
impl Patterns {
/// Parse TeX-style patterns: whitespace-separated, `%`/`#` start a comment line.
/// Parse patterns: whitespace-separated, `%` comments to the end of the line (`#` too at the
/// start of a line). A whole TeX pattern file works as is: `\patterns{…}` holds patterns and
/// `\hyphenation{…}` exception words written with their hyphens (`ni-gdy`); outside any
/// block, a token with a hyphen is an exception and anything else a pattern.
pub fn parse(text: &str) -> Patterns {
let mut p = Patterns::default();
let mut block = Block::Patterns;
for line in text.lines() {
let line = line.trim();
if line.is_empty() || line.starts_with('%') || line.starts_with('#') {
if line.starts_with('#') {
continue;
}
let line = line.split('%').next().unwrap_or("");
for tok in line.split_whitespace() {
p.insert(tok);
let mut tok = tok;
if let Some(cmd) = tok.strip_prefix('\\') {
let (name, rest) = cmd.split_once('{').unwrap_or((cmd, ""));
block = if name == "hyphenation" { Block::Exceptions } else { Block::Patterns };
tok = rest;
}
let closes = tok.ends_with('}');
let tok = tok.trim_end_matches('}');
if !tok.is_empty() {
if block == Block::Exceptions || tok.contains('-') {
p.insert_exception(tok);
} else {
p.insert(tok);
}
}
if closes {
block = Block::Patterns;
}
}
}
p
@@ -128,9 +177,40 @@ impl Patterns {
true
}
/// Add an exception word written with its hyphens (`ni-gdy`; no hyphen: never broken).
pub fn insert_exception(&mut self, word: &str) {
let mut plain = String::with_capacity(word.len());
let mut points = Vec::new();
let mut n = 0usize;
for c in word.chars() {
if c == '-' {
if n > 0 && points.last() != Some(&n) {
points.push(n);
}
} else {
plain.extend(c.to_lowercase());
n += 1;
}
}
points.retain(|&i| i < n);
if n > 0 {
self.exceptions.insert(plain, points.into_boxed_slice());
}
}
/// Break points of an exception word (lowercase), if it is one.
pub fn exception(&self, word: &str) -> Option<&[usize]> {
self.exceptions.get(word).map(|b| &b[..])
}
/// Number of exception words.
pub fn exception_count(&self) -> usize {
self.exceptions.len()
}
#[allow(dead_code)]
pub(crate) fn from_parts(alpha: Alphabet, map: FxMap<u64, Box<[u8]>>, max_len: usize) -> Patterns {
Patterns { alpha, map, max_len }
pub(crate) fn from_parts(alpha: Alphabet, map: FxMap<u128, Box<[u8]>>, max_len: usize) -> Patterns {
Patterns { alpha, map, max_len, exceptions: HashMap::new() }
}
/// Make `letters` part of the alphabet even if no pattern mentions them.
@@ -164,13 +244,13 @@ impl Patterns {
let mut vals = vec![0u8; codes.len() + 1];
apply(&self.map, self.max_len, &codes, &mut vals);
// Padded position p sits before padded char p; word char i is padded char i + 1.
Some(vals[1..codes.len()].to_vec())
vals.get(1..codes.len()).map(<[u8]>::to_vec)
}
/// Break points (char indices, a hyphen goes before that char) of a lowercase word.
pub fn points(&self, word: &[char]) -> Option<Vec<usize>> {
let v = self.values(word)?;
Some((1..word.len()).filter(|&i| v[i] % 2 == 1).collect())
Some((1..word.len()).filter(|&i| v.get(i).is_some_and(|x| x % 2 == 1)).collect())
}
/// The patterns in TeX notation (sorted by letters), e.g. for writing a pattern file.
@@ -183,17 +263,17 @@ impl Patterns {
.map
.iter()
.map(|(&key, vals)| {
let n = (key >> 58) as usize;
let n = ((key >> 120) as usize).min(MAX_PATTERN_LEN);
let letters: Vec<char> = (0..n).map(|i| ch(((key >> (6 * (n - 1 - i))) & 63) as u8)).collect();
let mut s = String::new();
for (i, l) in letters.iter().enumerate() {
if vals[i] > 0 {
s.push(char::from(b'0' + vals[i]));
if let Some(&v) = vals.get(i).filter(|v| **v > 0) {
s.push(char::from(b'0' + v.min(9)));
}
s.push(*l);
}
if vals[n] > 0 {
s.push(char::from(b'0' + vals[n]));
if let Some(&v) = vals.get(n).filter(|v| **v > 0) {
s.push(char::from(b'0' + v.min(9)));
}
(letters.into_iter().collect(), s)
})
@@ -203,18 +283,20 @@ impl Patterns {
}
}
/// Apply `map` to padded `codes`, raising `vals` (len = codes.len() + 1).
/// Apply `map` to padded `codes`, raising `vals` (len = codes.len() + 1; shorter is tolerated).
#[inline]
pub fn apply(map: &FxMap<u64, Box<[u8]>>, max_len: usize, codes: &[u8], vals: &mut [u8]) {
pub fn apply(map: &FxMap<u128, Box<[u8]>>, max_len: usize, codes: &[u8], vals: &mut [u8]) {
let n = codes.len();
for s in 0..n {
let mut k = 0u64;
for len in 1..=max_len.min(n - s) {
k = (k << 6) | codes[s + len - 1] as u64;
if let Some(v) = map.get(&(k | ((len as u64) << 58))) {
let mut k = 0u128;
for (len, &c) in codes.iter().skip(s).take(max_len.min(MAX_PATTERN_LEN)).enumerate() {
k = (k << 6) | c as u128;
if let Some(v) = map.get(&(k | (((len + 1) as u128) << 120))) {
for (j, &x) in v.iter().enumerate() {
if x > vals[s + j] {
vals[s + j] = x;
if let Some(slot) = vals.get_mut(s + j)
&& x > *slot
{
*slot = x;
}
}
}
@@ -237,4 +319,28 @@ mod tests {
assert!(tex.contains(&"hen5at".to_string()));
assert!(p.values(&['h', 'é']).is_none());
}
#[test]
fn tex_files_parse_with_blocks_comments_and_exceptions() {
let p = Patterns::parse(
"% a comment\n\\patterns{\n1na n2at % trailing comment\n.ab4cdefghijklm5n\n}\n\\hyphenation{\nni-gdy\nna-dal\nzawsze\n}\nhy-phen-ation\n",
);
assert_eq!(p.len(), 3);
assert_eq!(p.exception("nigdy"), Some(&[2][..]));
assert_eq!(p.exception("nadal"), Some(&[2][..]));
assert_eq!(p.exception("zawsze"), Some(&[][..]));
assert_eq!(p.exception("hyphenation"), Some(&[2, 6][..]));
assert_eq!(p.exception_count(), 4);
// A pattern longer than the old 9-letter limit applies.
let w: Vec<char> = "abcdefghijklmn".chars().collect();
assert_eq!(p.values(&w).unwrap()[13], 5);
assert!(p.to_tex().contains(&".ab4cdefghijklm5n".to_string()));
}
#[test]
fn apply_tolerates_short_value_buffers() {
let p = Patterns::parse("a1b");
let mut vals = [0u8; 1];
apply(&p.map, p.max_len, &[1, 2, 3, 1], &mut vals);
}
}
crates/proof/src/pl/grammar.rs
+542
View File
@@ -0,0 +1,542 @@
//! Polish grammar and punctuation checks: cheap, rule-based and tuned to stay quiet when unsure
//! (a missed comma costs less than a wrong mark on correct text). Messages are in Polish.
//!
//! - **Repeated word** (`się się`), except interjections and words Polish doubles on purpose
//! (`to to`, `tak tak`, `ha ha`).
//! - **Space before punctuation** (`tak ,`) and **double spaces**; emoticons (` :)`) and
//! ellipses are left alone.
//! - **Capital letter at the start of a sentence**, knowing Polish abbreviations (`np.`, `itd.`,
//! `r.`, `m.in.`), initials, ordinal numbers (`5. miejsce`), ellipses and dialogue dashes
//! (`– Tak? – zapytał`), after which lower case is correct.
//! - **Missing comma** before subordinating conjunctions and relative pronouns that always open a
//! clause: `że`, `ponieważ`, `gdyż`, `aby`, `żeby`, `jeśli`, `jeżeli`, `gdyby`, `lecz` and the
//! forms of `który` — unless a conjunction or particle comes first (`mimo że`, `chyba że`,
//! `tak aby`, `i który`) or, for `który`, a preposition (`w którym`: the comma goes before it).
//! - **`we`/`ze`** where the next word's consonant cluster requires them (`we wtorek`,
//! `ze szkoły`, `ze mną`).
//! - **Doubled punctuation** (`,,` — often a typed opening quote `„` — `..`, `;;`).
//!
//! No English rule (such as `a`/`an`) applies to Polish text.
use crate::{Issue, IssueKind, ProofLang, words};
/// Words Polish repeats on purpose: interjections, onomatopoeia and emphatic doubling.
const ALLOWED_REPEATS: &[&str] = &[
"to", "tak", "nie", "no", "ha", "he", "hi", "hej", "oj", "aj", "ej", "ach", "och", "ech", "puk", "hop", "bum", "cyk", "tik", "pa", "fiu", "la",
"bardzo", "dawno", "daleko", "długo",
];
/// Abbreviations written with a final period after which a sentence goes on in lower case
/// (lower case, without the period).
const ABBREVIATIONS: &[&str] = &[
"np", "itd", "itp", "tzn", "tj", "tzw", "ok", "ul", "al", "pl", "os", "in", "wg", "zob", "por", "str", "ss", "nr", "pkt", "ust", "art", "par",
"rozdz", "tab", "rys", "il", "przyp", "red", "wyd", "oprac", "tłum", "cyt", "jw", "ds", "im", "ks", "św", "bł", "prof", "dr", "doc", "hab",
"inż", "mgr", "lek", "płk", "gen", "mjr", "kpt", "ppłk", "sierż", "szer", "kmdr", "adm", "abp", "bp", "br", "ub", "bm", "pn", "płd", "płn",
"wsch", "zach", "gł", "godz", "min", "sek", "tys", "mln", "mld", "zł", "gr", "kg", "dag", "mies", "tyg", "pon", "wt", "śr", "czw", "pt", "sob",
"niedz", "sty", "lut", "mar", "kwi", "cze", "lip", "sie", "wrz", "paź", "lis", "gru", "ang", "niem", "franc", "łac", "ros", "wł", "hiszp", "pol",
"lit", "dosł", "przen", "pot", "zdrob", "fot", "ryc", "tel", "kom", "ew", "ob", "obyw", "pp", "zm", "ur", "ca", "vs", "etc", "cdn", "dot", "wz",
"zw", "zał", "cz", "rr", "ww", "jęz", "jun", "sen", "mec", "red", "dyr", "kier", "proc", "poj", "lp", "lm", "zaw", "woj", "pow", "gm", "wsp",
"nast", "poprz", "ang", "fr",
];
/// Subordinating conjunctions and relative pronouns that open a clause: a comma comes before
/// them.
const CLAUSE_OPENERS: &[&str] = &[
"że",
"ponieważ",
"gdyż",
"aby",
"żeby",
"jeśli",
"jeżeli",
"gdyby",
"lecz",
"który",
"która",
"które",
"którego",
"której",
"któremu",
"którą",
"którym",
"których",
"którymi",
"którzy",
];
/// Words after which a clause opener takes no comma of its own: conjunctions (`i że`), particles
/// (`tylko jeśli`, `zwłaszcza że`) and the first halves of compound conjunctions (`mimo że`,
/// `chyba że`, `tak aby`, `jak gdyby`, `na wypadek gdyby`, `z tym że`).
const NO_COMMA_AFTER: &[&str] = &[
"i",
"a",
"oraz",
"lub",
"albo",
"ani",
"czy",
"bądź",
"ale",
"lecz",
"bo",
"że",
"aby",
"żeby",
"by",
"niż",
"jak",
"jako",
"niby",
"tylko",
"nawet",
"właśnie",
"zwłaszcza",
"szczególnie",
"głównie",
"częściowo",
"jedynie",
"wyłącznie",
"chyba",
"mianowicie",
"przynajmniej",
"również",
"także",
"też",
"dopiero",
"jeszcze",
"już",
"zaś",
"więc",
"zatem",
"toteż",
"przecież",
"chociaż",
"choć",
"choćby",
"gdy",
"kiedy",
"jeśli",
"jeżeli",
"gdyby",
"skoro",
"zanim",
"dopóki",
"póki",
"aż",
"co",
"tak",
"mimo",
"pomimo",
"wtedy",
"teraz",
"dziś",
"dzisiaj",
"podczas",
"tym",
"bardziej",
"warunkiem",
"wypadek",
"zamiast",
"byle",
"lada",
"wiadomo",
"obojętnie",
"jedno",
"wszystkim",
"może",
"pewnie",
"prawdopodobnie",
"zapewne",
"dlatego",
"tyle",
"dość",
"ponieważ",
"gdyż",
"lecz",
"albowiem",
"oto",
"no",
];
/// Prepositions: a relative pronoun after one takes the comma before the preposition instead
/// (`dom, w którym`).
const PREPOSITIONS: &[&str] = &[
"w",
"we",
"z",
"ze",
"na",
"do",
"od",
"ode",
"o",
"u",
"po",
"za",
"przed",
"przede",
"przez",
"przeze",
"przy",
"nad",
"nade",
"pod",
"pode",
"dla",
"bez",
"beze",
"ku",
"między",
"pomiędzy",
"wśród",
"spośród",
"spod",
"spoza",
"sprzed",
"znad",
"zza",
"poza",
"ponad",
"poprzez",
"dzięki",
"według",
"wobec",
"oprócz",
"prócz",
"obok",
"koło",
"około",
"wokół",
"dookoła",
"naprzeciw",
"naprzeciwko",
"wzdłuż",
"wewnątrz",
"zewnątrz",
"względem",
"przeciw",
"przeciwko",
"wbrew",
"wskutek",
"śród",
"wedle",
"pośród",
"ponad",
"niedaleko",
"blisko",
];
fn issue(start: usize, end: usize, message: String, suggestions: Vec<String>) -> Issue {
Issue { start, end, kind: IssueKind::Grammar, message, suggestions, lang: ProofLang::Pl }
}
fn capitalize(w: &str) -> String {
crate::affix::capitalize(w)
}
/// Polish vowels (and the accented vowels of loanwords).
fn is_vowel(c: char) -> bool {
matches!(c.to_lowercase().next().unwrap_or(c), 'a' | 'ą' | 'e' | 'ę' | 'i' | 'o' | 'ó' | 'u' | 'y' | 'é' | 'á' | 'ö' | 'ü' | 'í' | 'ú')
}
fn is_consonant(c: char) -> bool {
c.is_alphabetic() && !is_vowel(c)
}
/// Grammar issues in a paragraph of Polish text, in order.
pub fn check(text: &str) -> Vec<Issue> {
let ws: Vec<(usize, usize)> = words(text);
let mut v = Vec::new();
repeated_words(text, &ws, &mut v);
spaces(text, &mut v);
sentence_capitals(text, &ws, &mut v);
missing_commas(text, &ws, &mut v);
we_ze(text, &ws, &mut v);
doubled_punctuation(text, &mut v);
v.sort_by_key(|i| (i.start, i.end));
v
}
fn repeated_words(text: &str, ws: &[(usize, usize)], v: &mut Vec<Issue>) {
for pair in ws.windows(2) {
let [(a0, b0), (a1, b1)] = pair else { continue };
let (Some(w0), Some(w1), Some(gap)) = (text.get(*a0..*b0), text.get(*a1..*b1), text.get(*b0..*a1)) else { continue };
if !gap.is_empty()
&& gap.chars().all(char::is_whitespace)
&& w0.chars().all(char::is_alphabetic)
&& w0.to_lowercase() == w1.to_lowercase()
&& !ALLOWED_REPEATS.contains(&w0.to_lowercase().as_str())
{
v.push(issue(*a0, *b1, format!("Powtórzone słowo: „{w1}”"), vec![w0.to_string()]));
}
}
}
fn spaces(text: &str, v: &mut Vec<Issue>) {
for (i, c) in text.char_indices() {
if c != ' ' {
continue;
}
let rest = text.get(i + 1..).unwrap_or("");
let mut it = rest.chars();
let Some(p) = it.next() else { continue };
let after = it.next();
let flag = match p {
',' => true,
// `;)` and `;-)` are emoticons.
';' => !matches!(after, Some(')' | '(' | '-' | 'P' | 'D')),
// ` ...` is an ellipsis, ` .5` a number.
'.' => !matches!(after, Some('.')) && !after.is_some_and(|a| a.is_ascii_digit()),
// ` :)`, ` :-)`, ` :D` and similar are emoticons; ` ?!` is still a stray space.
':' | '!' | '?' => after.is_none_or(|a| a.is_whitespace() || matches!(a, '"' | '”' | '»' | '’' | '!' | '?')),
_ => false,
};
if flag {
v.push(issue(i, i + 1 + p.len_utf8(), "Zbędna spacja przed znakiem interpunkcyjnym".into(), vec![p.to_string()]));
}
}
for (i, _) in text.match_indices(" ") {
if i > 0 && text.get(..i).is_some_and(|t| t.ends_with(' ')) {
continue;
}
v.push(issue(i, i + 2, "Podwójna spacja".into(), vec![" ".into()]));
}
}
/// Is `prev`, before a period, an abbreviation, an initial, a number or a Roman numeral (after
/// which a lower-case word doesn't start a sentence)?
fn abbreviation_or_number(prev: &str) -> bool {
prev.chars().count() <= 1
|| prev.chars().any(|c| c.is_ascii_digit())
|| (prev.chars().all(|c| matches!(c, 'I' | 'V' | 'X' | 'L' | 'C' | 'D' | 'M')))
|| ABBREVIATIONS.contains(&prev.to_lowercase().as_str())
}
const CLOSING: [char; 8] = ['"', '”', '’', '\'', ')', '»', ']', '\u{1}'];
const OPENING: [char; 8] = ['„', '"', '“', '«', '(', '[', '\'', '‚'];
/// Does the gap between two words end a sentence: a `.`, `!` or `?` (not an ellipsis, not after
/// an abbreviation), then space, with nothing but quotes and brackets around?
fn ends_sentence(gap: &str, prev: &str) -> bool {
let t = gap.trim_start_matches(CLOSING);
let Some(end) = t.chars().next() else { return false };
if !matches!(end, '.' | '!' | '?') || t.starts_with("..") {
return false;
}
if end == '.' && abbreviation_or_number(prev) {
return false;
}
let rest = t.trim_start_matches(['.', '!', '?']).trim_start_matches(CLOSING);
rest.starts_with(char::is_whitespace) && rest.trim_start().chars().all(|c| OPENING.contains(&c))
}
fn sentence_capitals(text: &str, ws: &[(usize, usize)], v: &mut Vec<Issue>) {
// List items (`– jabłka,`) may start in lower case.
let list_item = text.trim_end().ends_with([',', ';']);
for (k, &(a, b)) in ws.iter().enumerate() {
let Some(w) = text.get(a..b) else { continue };
let starts = if k == 0 {
let before = text.get(..a).unwrap_or("");
!list_item && before.chars().all(|c| c.is_whitespace() || OPENING.contains(&c))
} else {
let Some(&(pa, pb)) = ws.get(k - 1) else { continue };
ends_sentence(text.get(pb..a).unwrap_or(""), text.get(pa..pb).unwrap_or(""))
};
if starts && w.chars().next().is_some_and(char::is_lowercase) && w.chars().all(char::is_alphabetic) {
v.push(issue(a, b, "Wielka litera na początku zdania".into(), vec![capitalize(w)]));
}
}
}
fn missing_commas(text: &str, ws: &[(usize, usize)], v: &mut Vec<Issue>) {
for k in 1..ws.len() {
let (Some(&(pa, pb)), Some(&(a, b))) = (ws.get(k - 1), ws.get(k)) else { continue };
let (Some(prev), Some(w), Some(gap)) = (text.get(pa..pb), text.get(a..b), text.get(pb..a)) else { continue };
if !CLAUSE_OPENERS.contains(&w) || gap.is_empty() || !gap.chars().all(|c| c == ' ' || c == '\u{a0}') {
continue;
}
if !prev.chars().all(char::is_alphabetic) {
continue;
}
let p = prev.to_lowercase();
let p = p.as_str();
if NO_COMMA_AFTER.contains(&p) {
continue;
}
if w.starts_with("któr") && (PREPOSITIONS.contains(&p) || p == "to") {
continue;
}
v.push(issue(pb, b, format!("Brak przecinka przed „{w}”"), vec![format!(", {w}")]));
}
}
fn we_ze(text: &str, ws: &[(usize, usize)], v: &mut Vec<Issue>) {
for pair in ws.windows(2) {
let [(a0, b0), (a1, b1)] = pair else { continue };
let (Some(w0), Some(w1), Some(gap)) = (text.get(*a0..*b0), text.get(*a1..*b1), text.get(*b0..*a1)) else { continue };
if gap != " " && gap != "\u{a0}" {
continue;
}
let next = w1.to_lowercase();
let c: Vec<char> = next.chars().collect();
let at = |i: usize| c.get(i).copied();
let (long, wanted) = match w0 {
"w" | "W" => ("we", (at(0) == Some('w') && at(1).is_some_and(is_consonant)) || next == "mnie"),
"z" | "Z" => {
// The first sound: `sz` is one consonant (`z szafy`, but `ze szkoły`).
let after = if at(0) == Some('s') && at(1) == Some('z') { at(2) } else { at(1) };
let sibilant = matches!(at(0), Some('s' | 'z' | 'ś' | 'ź' | 'ż'));
("ze", (sibilant && after.is_some_and(is_consonant)) || next == "mną")
}
_ => continue,
};
if wanted {
let fixed = if w0.starts_with(char::is_uppercase) { capitalize(long) } else { long.to_string() };
v.push(issue(*a0, *b0, format!("Przed tym słowem piszemy „{long}”"), vec![fixed]));
}
}
}
fn doubled_punctuation(text: &str, v: &mut Vec<Issue>) {
let bytes = text.as_bytes();
let mut i = 0;
while let Some(&b) = bytes.get(i) {
if !matches!(b, b',' | b'.' | b';') {
i += 1;
continue;
}
let run = bytes.get(i..).map_or(0, |r| r.iter().take_while(|&&x| x == b).count());
let flag = match b {
b'.' => run == 2,
_ => run >= 2,
};
if flag {
let end = i + run;
let suggestions: Vec<String> = match b {
// `,,` before a word, after a space: a typed opening quote.
b',' if text.get(end..).and_then(|t| t.chars().next()).is_some_and(char::is_alphabetic)
&& (i == 0 || text.get(..i).is_some_and(|t| t.ends_with(char::is_whitespace))) =>
{
vec!["„".into()]
}
b'.' => vec![".".into(), "…".into()],
_ => vec![char::from(b).to_string()],
};
v.push(issue(i, end, "Powtórzony znak interpunkcyjny".into(), suggestions));
}
i += run.max(1);
}
}
#[cfg(test)]
mod tests {
use super::*;
fn messages(t: &str) -> Vec<String> {
check(t).into_iter().map(|i| format!("{}@{}", i.message, &t[i.start..i.end])).collect()
}
fn clean(t: &str) {
let m = messages(t);
assert!(m.is_empty(), "{t:?}: {m:?}");
}
#[test]
fn repeated_words() {
let v = check("Ala ma ma kota i się się cieszy.");
assert_eq!(v.len(), 2, "{v:?}");
assert_eq!(v[0].message, "Powtórzone słowo: „ma”");
assert_eq!(v[0].suggestions, ["ma"]);
assert!(v.iter().all(|i| i.lang == ProofLang::Pl && i.kind == IssueKind::Grammar));
clean("Tak tak, to to samo. Ha ha!");
}
#[test]
fn spaces_and_punctuation() {
let m = messages("Ala ma kota , a kot ma Alę . Czy na pewno ?");
assert!(m.contains(&"Zbędna spacja przed znakiem interpunkcyjnym@ ,".to_string()), "{m:?}");
assert!(m.contains(&"Zbędna spacja przed znakiem interpunkcyjnym@ .".to_string()), "{m:?}");
assert!(m.contains(&"Zbędna spacja przed znakiem interpunkcyjnym@ ?".to_string()), "{m:?}");
assert!(m.contains(&"Podwójna spacja@ ".to_string()), "{m:?}");
clean("Fajnie :) Naprawdę :-) Albo :D Wartość .5 i wielokropek ...");
let m = messages("Powiedział ,,tak” i poszedł.. Potem;; koniec.");
assert_eq!(m.iter().filter(|x| x.starts_with("Powtórzony znak")).count(), 3, "{m:?}");
assert_eq!(check("Powiedział ,,tak”.").iter().find(|i| i.message.starts_with("Powtórzony")).unwrap().suggestions, ["„"]);
clean("Czekał… i czekał... aż do rana.");
}
#[test]
fn sentence_start_capitals() {
let m = messages("to jest zdanie. a to drugie! czy trzecie? tak.");
assert_eq!(m.iter().filter(|x| x.starts_with("Wielka litera")).count(), 4, "{m:?}");
assert_eq!(check("ala ma kota.")[0].suggestions, ["Ala"]);
// Abbreviations, initials, ordinals, dates, ellipses and dialogue dashes.
clean("Kupiłem owoce, np. jabłka, gruszki itp. oraz warzywa.");
clean("W 1990 r. powstała firma, m.in. dzięki dotacji.");
clean("Spotkałem J. Kowalskiego i prof. Nowaka przy ul. Długiej.");
clean("Zajął 5. miejsce w XX w. sporcie.");
clean("Czekałem... aż w końcu przyszedł.");
clean("– Naprawdę? – zapytał. – Tak – odpowiedziała.");
clean("Powiedział: „Wracam”. Potem wyszedł.");
// A list item may start in lower case.
clean("jabłka,");
}
#[test]
fn missing_commas() {
let m = messages("Wiem że przyjdzie. Mam kota który śpi. Zostałem bo musiałem. Idę aby zdążyć. Nie przyszedł ponieważ padało.");
for want in [
"Brak przecinka przed „że”@ że",
"Brak przecinka przed „który”@ który",
"Brak przecinka przed „aby”@ aby",
"Brak przecinka przed „ponieważ”@ ponieważ",
] {
assert!(m.contains(&want.to_string()), "{want}: {m:?}");
}
assert_eq!(check("Wiem że tak.")[0].suggestions, [", że"]);
// Compound conjunctions, particles, conjunctions and prepositions take no comma there.
clean("Przyszedł, mimo że padało, chyba że nie.");
clean("Zrobię to, tylko jeśli zdążę, i to tak aby nikt nie widział.");
clean("To dom, w którym mieszkam, i ogród, przez który idę.");
clean("Wiem, że przyjdzie, i że zostanie.");
clean("Myślę, że gdyby przyszedł, byłoby dobrze.");
clean("Zachowywał się, jak gdyby nigdy nic.");
clean("Który z nich? To który wybierasz?");
clean("Po raz nie wiadomo który.");
clean("Dlatego że padało, zostaliśmy.");
}
#[test]
fn we_and_ze() {
let m = messages("Spotkamy się w wtorek w Wrocławiu z szkoły z mną.");
assert_eq!(m.iter().filter(|x| x.starts_with("Przed tym słowem")).count(), 4, "{m:?}");
let v = check("Z szkoły wracam.");
assert_eq!(v[0].suggestions, ["Ze"]);
clean("Byłem w wodzie, w wieku dziesięciu lat, z szafą, z siostrą i z zamkiem.");
clean("Wracam ze szkoły we wtorek ze mną.");
}
#[test]
fn correct_polish_text_is_clean() {
clean(
"Zażółć gęślą jaźń. Chrząszcz brzmi w trzcinie, a źdźbło trawy się kołysze. \
Przyjdę jutro, chyba że będzie padać. Książka, którą czytam, jest ciekawa.",
);
clean("Pięćdziesięciu uczniów przyszło, ponieważ ogłoszono konkurs.");
}
proptest::proptest! {
#[test]
fn never_panics(s in "\\PC{0,80}") {
let _ = check(&s);
}
#[test]
fn never_panics_on_polish_like_text(s in "[a-ząćęłńóśźżA-Z ,.;:!?„”…\\-]{0,60}") {
for i in check(&s) {
proptest::prop_assert!(i.start <= i.end && i.end <= s.len());
proptest::prop_assert!(s.is_char_boundary(i.start) && s.is_char_boundary(i.end));
}
}
}
}
crates/proof/src/pl/mod.rs
+204
View File
@@ -0,0 +1,204 @@
//! Polish proofing.
//!
//! - **Spelling** against the SJP.PL Hunspell dictionary installed on the system
//! ([`dictionary`], found by [`crate::hunspell`]: `pl_PL.aff`/`pl_PL.dic` from the OS, a
//! package manager or LibreOffice; it isn't bundled). Without it Polish text gets no spelling
//! marks. Capitalized stems are names (`Warszawa`); a sentence may start with any word and
//! headings may be in capitals. Hyphenated compounds are checked part by part
//! (`biało-czerwony`), foreign names inflected with an apostrophe are accepted
//! (`Kennedy'ego`), and the user dictionary applies as in English.
//! - **Suggestions** tuned for Polish ([`suggest()`]): missing diacritics first (`zolw` → `żółw`), then
//! the common spelling confusions (`ż`/`rz`, `u`/`ó`, `h`/`ch`…), a missing space (`napewno` →
//! `na pewno`), one-letter typos, and a dictionary scan for anything further off.
//! - **Grammar** ([`grammar`]): cheap, reliable rules with Polish messages.
pub mod grammar;
mod suggest;
#[cfg(test)]
pub(crate) mod test_dict;
use std::sync::{Arc, RwLock};
use crate::affix::AffixDict;
pub use suggest::suggest;
/// The Polish dictionary: `None` until first use, then the installed one (or an empty one).
static DICT: RwLock<Option<Arc<AffixDict>>> = RwLock::new(None);
/// File names of a Polish Hunspell dictionary.
const NAMES: &[&str] = &["pl_PL", "pl-PL", "pl"];
/// The Polish dictionary: the installed Hunspell `pl_PL` (read on first use), one given with
/// [`install_dictionary`], or an empty one when none is available (Polish spelling is off then;
/// see [`has_dictionary`]).
pub fn dictionary() -> Arc<AffixDict> {
if let Some(d) = DICT.read().unwrap_or_else(|e| e.into_inner()).as_ref() {
return d.clone();
}
let loaded = Arc::new(load_installed());
let mut slot = DICT.write().unwrap_or_else(|e| e.into_inner());
// Another thread (or an install) may have filled it meanwhile: that one wins.
slot.get_or_insert(loaded).clone()
}
fn load_installed() -> AffixDict {
let Some((aff, dic)) = crate::hunspell::find(NAMES) else {
log::info!("Polish spelling: no Hunspell pl_PL dictionary installed");
return AffixDict::default();
};
crate::hunspell::load(&aff, &dic).unwrap_or_else(|e| {
log::warn!("Polish spelling: {e}");
AffixDict::default()
})
}
/// Use this Polish dictionary from now on (Hunspell `.aff` and `.dic` text, decoded to UTF-8),
/// for example one the user picked. On error the current dictionary stays.
pub fn install_dictionary(aff: &str, dic: &str) -> Result<(), String> {
let d = AffixDict::from_hunspell(aff, dic, &[])?;
*DICT.write().unwrap_or_else(|e| e.into_inner()) = Some(Arc::new(d));
Ok(())
}
/// Forget the loaded dictionary, so the next check looks for an installed one again (after
/// [`crate::hunspell::set_dictionary_dir`] or a dictionary was installed).
pub fn reload_dictionary() {
*DICT.write().unwrap_or_else(|e| e.into_inner()) = None;
}
/// Is a Polish dictionary available (spelling is checked)?
pub fn has_dictionary() -> bool {
!dictionary().is_empty()
}
/// Endings Polish adds after an apostrophe to foreign names whose last letter isn't pronounced
/// (`Kennedy'ego`, `Wilde'a`, `Shakespeare'owi`).
const APOSTROPHE_ENDINGS: &[&str] = &["a", "ach", "ami", "e", "ego", "em", "emu", "i", "m", "om", "owi", "owie", "u", "y", "ów", "ą", "ę"];
/// Is `c` a letter of the Latin script (the only one the Polish dictionary covers)?
fn is_latin_letter(c: char) -> bool {
c.is_alphabetic() && (c < '\u{0250}' || ('\u{1E00}'..='\u{1EFF}').contains(&c))
}
/// Is `word` spelled correctly in Polish? Numbers, single letters, short words in capitals
/// (acronyms: `PKO`, `ZUS`), words with digits and words in other scripts are accepted, and so
/// is every word when no Polish dictionary is installed.
pub fn is_correct(word: &str) -> bool {
is_correct_with(&dictionary(), word)
}
/// [`is_correct`] against a given dictionary (an empty one accepts everything).
pub fn is_correct_with(d: &AffixDict, word: &str) -> bool {
if d.is_empty() {
return true; // no Polish dictionary installed: no spelling marks
}
let w = word.trim_matches(|c: char| !c.is_alphanumeric() && c != '\'' && c != '’');
let w = w.trim_matches(['\'', '’']);
let n = w.chars().count();
if n <= 1 || w.chars().any(|c| c.is_ascii_digit()) || !w.chars().any(char::is_alphabetic) {
return true;
}
if w.chars().all(|c| !c.is_lowercase()) && n <= 6 {
return true;
}
if !w.chars().any(is_latin_letter) {
return true; // other scripts: no dictionary
}
let w = w.replace('’', "'");
if w.contains('-') {
return w.split('-').all(|p| p.chars().count() <= 1 || known(d, p));
}
known(d, &w)
}
/// Is one word (no hyphens) in the dictionary, the user dictionary, or a foreign name with a
/// Polish ending after an apostrophe?
fn known(d: &AffixDict, w: &str) -> bool {
if d.check(w) || crate::in_user_dictionary(&w.to_lowercase()) {
return true;
}
if let Some((stem, ending)) = w.split_once('\'') {
return stem.chars().next().is_some_and(char::is_uppercase) && stem.chars().count() >= 2 && APOSTROPHE_ENDINGS.contains(&ending);
}
false
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_dictionary_loads() {
test_dict::install();
let d = dictionary();
assert!(has_dictionary());
assert!(d.len() > 50, "{}", d.len());
assert_eq!(d.rule_count(), 5, "the test dictionary's affix rules");
assert!(!d.rep().is_empty() && !d.map().is_empty() && !d.try_chars().is_empty());
}
#[test]
fn without_a_dictionary_nothing_is_flagged_and_nothing_suggested() {
let none = AffixDict::default();
for w in ["żeka", "kturzy", "xqzv", ""] {
assert!(is_correct_with(&none, w), "{w}");
}
assert!(suggest::suggest_in(&none, "żeka", 5).is_empty());
assert!(install_dictionary("FLAG long\n", "1\nkot\n").is_err(), "unsupported files are refused");
}
#[test]
fn polish_words_are_correct() {
test_dict::install();
for w in [
"źdźbło",
"gżegżółka",
"chrząszcz",
"pięćdziesięciu",
"rzeka",
"którzy",
"żółw",
"Zażółć",
"gęślą",
"jaźń",
"konstytucja",
"konstytucyjnego",
"niebieskiego",
"nieładnie",
"przyszliśmy",
"zrobilibyśmy",
"Warszawa",
"Warszawie",
"WARSZAWA",
"Kot",
"biało-czerwony",
"polsko-niemieckiej",
"Kennedy'ego",
"Shakespeare’a",
"PKO",
"2026",
"e-mail",
"w",
"się",
] {
assert!(is_correct(w), "{w}");
}
}
#[test]
fn misspellings_are_flagged() {
test_dict::install();
for w in ["żeka", "kturzy", "napewno", "wogóle", "zolw", "chuśtawka", "bżoza", "warszawa", "niewiem", "pszyjaciel", "Kennedy'xyz"] {
assert!(!is_correct(w), "{w}");
}
}
#[test]
fn user_dictionary_applies() {
test_dict::install();
assert!(!is_correct("zxqpolskisłowo"));
crate::add_word("Zxqpolskisłowo");
assert!(is_correct("zxqpolskisłowo"));
}
}
crates/proof/src/pl/suggest.rs
+437
View File
@@ -0,0 +1,437 @@
//! Polish spelling suggestions, best first.
//!
//! Candidates come in rounds of rising cost, each checked against the dictionary:
//! 1. letter case (`warszawa` → `Warszawa`);
//! 2. missing or wrong diacritics, from the dictionary's `MAP` groups (`zolw` → `żółw`, `kturzy`
//! → `którzy`), up to four letters changed — Polish typed without diacritics is the most
//! common slip, so adding them ranks first, while taking away a diacritic the writer typed
//! ranks after the known confusions;
//! 3. the dictionary's `REP` table of common confusions (`żeka` → `rzeka`, `chuśtawka` →
//! `huśtawka`, `pszyjaciel` → `przyjaciel`);
//! 4. a missing space (`napewno` → `na pewno`, `wogóle` → `w ogóle`, `kochamcię` → `kocham
//! cię`), cheaper when one half is a short function word;
//! 5. one typo: a letter left out, added, swapped with the next or mistyped (`TRY` letters);
//! 6. when that finds fewer than asked for: the stems that start like the word (diacritics
//! folded) and share the most letter pairs with it, whose forms are ranked by an edit distance
//! in which a diacritic slip costs half (two typos at most).
//!
//! The input is bounded (40 letters), and so is every round, so a call costs a few thousand
//! dictionary lookups at most.
use std::collections::{HashMap, HashSet};
use crate::affix::{AffixDict, Casing, capitalize, casing};
/// Short words often run into the next one (`na pewno`, `w ogóle`, `nie wiem`, `przede
/// wszystkim`).
const LEADING_WORDS: &[&str] = &[
"a", "i", "o", "u", "w", "z", "we", "ze", "na", "po", "do", "od", "za", "co", "to", "nie", "jak", "tak", "przy", "bez", "pod", "nad", "dla",
"że", "by", "czy", "już", "ani", "przed", "przez", "przede", "ku", "aż", "bo", "ten", "ta", "te", "tym", "tej", "też", "jest", "ode", "beze",
"nade",
];
/// Short words often run into the one before (`kocham cię`, `daj mi`, `bałem się`).
const TRAILING_WORDS: &[&str] =
&["się", "cię", "mi", "mu", "ci", "go", "ją", "je", "nam", "wam", "im", "by", "że", "no", "mnie", "tobie", "jej", "ich"];
const MAX_LETTERS: usize = 40;
const MAX_MAP_CHANGES: usize = 4;
const MAX_MAP_VARIANTS: usize = 4096;
/// Stems the scan of round 6 expands.
const SCAN_STEMS: usize = 48;
const COST_CASE: u32 = 0;
const COST_ADD_DIACRITIC: u32 = 4;
const COST_OTHER_MAP: u32 = 7;
const COST_DROP_DIACRITIC: u32 = 11;
const COST_REP: u32 = 10;
const COST_SPLIT: u32 = 12;
const COST_SPLIT_RARE: u32 = 18;
const COST_SWAP: u32 = 14;
const COST_DROP_OR_ADD: u32 = 15;
const COST_MISTYPE: u32 = 16;
const COST_SCAN: u32 = 20;
struct Collector<'a> {
d: &'a AffixDict,
/// The word, lower case.
input: String,
tested: HashSet<String>,
/// Suggestion → (cost, order found).
found: HashMap<String, (u32, usize)>,
seq: usize,
}
impl Collector<'_> {
/// Check a lower-case candidate (a word, or two separated by a space) and keep it if every
/// word is in the dictionary, as written or capitalized (names).
fn offer(&mut self, cand: &str, cost: u32) {
if cand.is_empty() || cand == self.input || !self.tested.insert(cand.to_string()) {
return;
}
let mut words = Vec::new();
let mut cost = cost;
for part in cand.split(' ') {
if self.d.contains(part) {
words.push(part.to_string());
} else {
let c = capitalize(part);
if !self.d.contains(&c) {
return;
}
words.push(c);
// A name ranks after a common word at the same distance (`moze` → `może`, then
// `Mozę`).
cost += 1;
}
}
self.keep(words.join(" "), cost);
}
/// Keep a suggestion known to be correct.
fn keep(&mut self, text: String, cost: u32) {
self.seq += 1;
let seq = self.seq;
let e = self.found.entry(text).or_insert((cost, seq));
if cost < e.0 {
*e = (cost, seq);
}
}
fn count_within(&self, cost: u32) -> usize {
self.found.values().filter(|(c, _)| *c <= cost).count()
}
}
/// The MAP group of each letter (its other members are its likely confusions).
fn alternatives(c: char, map: &[Vec<char>]) -> Vec<char> {
map.iter().filter(|g| g.contains(&c)).flat_map(|g| g.iter().copied()).filter(|&x| x != c).collect()
}
/// The cost of changing letter `from` to its MAP relative `to`. Adding a missing diacritic is the
/// commonest slip (Polish typed on a keyboard without Polish letters: `zolw`); a diacritic the
/// writer did type is likelier right, so taking one away costs more than a known confusion from
/// the REP table (`żeka` → `rzeka` before `zeka`).
fn map_cost(from: char, to: char) -> u32 {
match (from.is_ascii(), to.is_ascii()) {
(true, false) => COST_ADD_DIACRITIC,
(false, true) => COST_DROP_DIACRITIC,
_ => COST_OTHER_MAP,
}
}
/// Every way of changing up to [`MAX_MAP_CHANGES`] letters to a MAP relative, with its cost.
fn map_variants(chars: &[char], alts: &[Vec<char>], i: usize, changes: usize, cost: u32, buf: &mut Vec<char>, out: &mut Vec<(String, u32)>) {
if out.len() >= MAX_MAP_VARIANTS {
return;
}
let Some(&c) = chars.get(i) else {
if changes > 0 {
out.push((buf.iter().collect(), cost));
}
return;
};
buf.push(c);
map_variants(chars, alts, i + 1, changes, cost, buf, out);
buf.pop();
if changes < MAX_MAP_CHANGES {
for &a in alts.get(i).map(Vec::as_slice).unwrap_or_default() {
buf.push(a);
map_variants(chars, alts, i + 1, changes + 1, cost + map_cost(c, a), buf, out);
buf.pop();
}
}
}
/// A letter's MAP representative (the first of its group), lower case: `ż` → `z`, `ó` → `o`.
fn fold(c: char, map: &[Vec<char>]) -> char {
let l = c.to_lowercase().next().unwrap_or(c);
map.iter().find(|g| g.contains(&l)).and_then(|g| g.first().copied()).unwrap_or(l)
}
/// Optimal string alignment distance where a MAP slip (`a`/`ą`, `u`/`ó`) costs 1 and any other
/// edit 2; `limit + 1` once it is certainly above `limit`.
fn weighted_distance(a: &[char], b: &[char], related: &dyn Fn(char, char) -> bool, limit: u32) -> u32 {
let m = b.len();
let over = limit + 1;
let mut prev2 = vec![0u32; m + 1];
let mut prev: Vec<u32> = (0..=m as u32).map(|j| j * 2).collect();
let mut cur = vec![0u32; m + 1];
for (i, &ai) in a.iter().enumerate() {
if let Some(c) = cur.get_mut(0) {
*c = (i as u32 + 1) * 2;
}
let mut row_min = u32::MAX;
for (j, &bj) in b.iter().enumerate() {
let sub = if ai == bj {
0
} else if related(ai, bj) {
1
} else {
2
};
let at = |v: &[u32], k: usize| v.get(k).copied().unwrap_or(u32::MAX / 4);
let mut v = (at(&prev, j + 1) + 2).min(at(&cur, j) + 2).min(at(&prev, j) + sub);
if i > 0 && j > 0 && a.get(i - 1) == Some(&bj) && b.get(j - 1) == Some(&ai) {
v = v.min(at(&prev2, j - 1) + 2);
}
if let Some(c) = cur.get_mut(j + 1) {
*c = v;
}
row_min = row_min.min(v);
}
if m > 0 && row_min > limit {
return over;
}
std::mem::swap(&mut prev2, &mut prev);
std::mem::swap(&mut prev, &mut cur);
}
prev.get(m).copied().unwrap_or(over).min(over)
}
/// Spelling suggestions for a Polish word, best first, at most `max`, in the word's letter case.
pub fn suggest(word: &str, max: usize) -> Vec<String> {
suggest_in(&super::dictionary(), word, max)
}
/// [`suggest`] against a given dictionary.
pub fn suggest_in(d: &AffixDict, word: &str, max: usize) -> Vec<String> {
let word = word.trim();
let n = word.chars().count();
if n == 0 || n > MAX_LETTERS || max == 0 || d.is_empty() || word.contains(char::is_whitespace) {
return Vec::new();
}
let case = casing(word);
let input = word.to_lowercase();
let chars: Vec<char> = input.chars().collect();
let mut col = Collector { d, input: input.clone(), tested: HashSet::new(), found: HashMap::new(), seq: 0 };
// 1. Letter case.
if d.contains(&input) && word != input {
col.keep(input.clone(), COST_CASE);
} else if d.contains(&capitalize(&input)) && word != capitalize(&input) {
col.keep(capitalize(&input), COST_CASE);
}
// 2. Diacritics and other MAP slips.
let map = d.map();
let alts: Vec<Vec<char>> = chars.iter().map(|&c| alternatives(c, map)).collect();
let mut variants = Vec::new();
map_variants(&chars, &alts, 0, 0, 0, &mut Vec::with_capacity(n), &mut variants);
variants.sort_by_key(|v| v.1);
for (v, cost) in &variants {
col.offer(v, *cost);
}
// 3. Common confusions.
for (from, to) in d.rep() {
if from.is_empty() {
continue;
}
for (i, _) in input.match_indices(from.as_str()) {
if let (Some(a), Some(b)) = (input.get(..i), input.get(i + from.len()..)) {
col.offer(&format!("{a}{to}{b}"), COST_REP);
}
}
}
// 4. A missing space: cheap after a short leading word or before a short trailing one,
// dearer between two longer words, never leaving a stray short half (`że ka`).
for (i, _) in input.char_indices().skip(1) {
let (Some(a), Some(b)) = (input.get(..i), input.get(i..)) else { continue };
let long = |p: &str| p.chars().count() >= 3;
let cost = if (LEADING_WORDS.contains(&a) && long(b)) || (long(a) && TRAILING_WORDS.contains(&b)) {
COST_SPLIT
} else if long(a) && long(b) {
COST_SPLIT_RARE
} else {
continue;
};
col.offer(&format!("{a} {b}"), cost);
}
// 5. One typo.
let mut letters: Vec<char> = d.try_chars().chars().filter(|c| c.is_lowercase()).collect();
if letters.is_empty() {
letters = "aioeznrwcysptkmdłuljągbhęśćóżfńźvqx".chars().collect();
}
let s = |v: &[char]| v.iter().collect::<String>();
for i in 0..n.saturating_sub(1) {
if chars.get(i) != chars.get(i + 1) {
let mut v = chars.clone();
v.swap(i, i + 1);
col.offer(&s(&v), COST_SWAP);
}
}
for i in 0..n {
let mut v = chars.clone();
v.remove(i);
col.offer(&s(&v), COST_DROP_OR_ADD);
}
for i in 0..=n {
for &c in &letters {
let mut v = chars.clone();
v.insert(i, c);
col.offer(&s(&v), COST_DROP_OR_ADD);
}
}
for i in 0..n {
for &c in &letters {
if chars.get(i) != Some(&c) {
let mut v = chars.clone();
if let Some(x) = v.get_mut(i) {
*x = c;
}
col.offer(&s(&v), COST_MISTYPE);
}
}
}
// 6. Further off: scan the stems that start like the word.
if col.count_within(COST_MISTYPE) < max {
scan(&mut col, &chars);
}
let mut found: Vec<(String, (u32, usize))> = col.found.into_iter().collect();
found.sort_by_key(|(_, k)| *k);
let mut out: Vec<String> = Vec::new();
for (text, _) in found {
let t = match case {
Casing::Initial => capitalize(&text),
Casing::Upper if n > 1 => text.to_uppercase(),
_ => text,
};
if t != word && !out.contains(&t) {
out.push(t);
}
if out.len() >= max {
break;
}
}
out
}
/// Round 6: rank the stems starting with the word's first letter (or a MAP relative of it, either
/// case) by shared letter pairs, then keep the forms of the best within two typos.
fn scan(col: &mut Collector, chars: &[char]) {
let d = col.d;
let map = d.map();
let Some(&first) = chars.first() else { return };
let folded: Vec<char> = chars.iter().map(|&c| fold(c, map)).collect();
let pairs: HashSet<(char, char)> = std::iter::once(' ').chain(folded.iter().copied()).zip(folded.iter().copied()).collect();
let mut firsts: Vec<char> = std::iter::once(first).chain(alternatives(first, map)).collect();
let upper: Vec<char> = firsts.iter().flat_map(|c| c.to_uppercase()).collect();
firsts.extend(upper);
firsts.sort_unstable();
firsts.dedup();
let n = folded.len();
let mut best: Vec<(u32, &str)> = Vec::new();
for f in firsts {
let prefix = f.to_string();
for stem in d.stems_with_prefix(&prefix) {
let sf: Vec<char> = stem.chars().map(|c| fold(c, map)).collect();
let shared = std::iter::once(' ').chain(sf.iter().copied()).zip(sf.iter().copied()).filter(|p| pairs.contains(p)).count();
if shared < 2 {
continue;
}
// Shared pairs, less the stem's letters the word doesn't explain, less a length gap
// beyond what an ending explains.
let extra = sf.len().saturating_sub(shared);
let gap = sf.len().abs_diff(n).saturating_sub(4);
let score = (shared * 4).saturating_sub(extra * 3 + gap * 2) as u32;
if score == 0 {
continue;
}
if best.len() < SCAN_STEMS {
best.push((score, stem));
} else if let Some(min) = best.iter_mut().min_by_key(|b| b.0)
&& score > min.0
{
*min = (score, stem);
}
}
}
best.sort_by(|a, b| b.0.cmp(&a.0).then(a.1.cmp(b.1)));
let related = |a: char, b: char| fold(a, map) == fold(b, map);
let limit = 4;
for (_, stem) in best {
for form in d.forms(stem) {
let lf: Vec<char> = form.to_lowercase().chars().collect();
if lf.len().abs_diff(chars.len()) > 2 {
continue;
}
let dist = weighted_distance(&lf, chars, &related, limit);
if dist <= limit && lf.as_slice() != chars {
col.keep(form, COST_SCAN + dist * 3);
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
fn has(word: &str, want: &str) {
let s = suggest(word, 8);
assert!(s.iter().any(|x| x == want), "{word} → {s:?}, want {want}");
}
#[test]
fn polish_misspellings_get_the_right_suggestion() {
crate::pl::test_dict::install();
has("żeka", "rzeka");
has("kturzy", "którzy");
has("napewno", "na pewno");
has("wogóle", "w ogóle");
has("niewiem", "nie wiem");
has("zolw", "żółw");
has("gzegzolka", "gżegżółka");
has("chuśtawka", "huśtawka");
has("bżoza", "brzoza");
has("pszyjaciel", "przyjaciel");
has("warszawa", "Warszawa");
has("Żeka", "Rzeka");
has("ZOLW", "ŻÓŁW");
has("konstytucija", "konstytucja");
has("wziąść", "wziąć");
has("dziewczyan", "dziewczyna");
}
#[test]
fn the_likeliest_fix_comes_first() {
crate::pl::test_dict::install();
assert_eq!(suggest("kturzy", 3).first().map(String::as_str), Some("którzy"));
assert_eq!(suggest("zolw", 3).first().map(String::as_str), Some("żółw"));
assert_eq!(suggest("napewno", 3).first().map(String::as_str), Some("na pewno"));
assert_eq!(suggest("żeka", 3).first().map(String::as_str), Some("rzeka"));
}
#[test]
fn edge_cases_stay_empty_or_bounded() {
crate::pl::test_dict::install();
assert!(suggest("", 5).is_empty());
assert!(suggest("kot", 0).is_empty());
assert!(suggest(&"a".repeat(41), 5).is_empty());
assert!(suggest("dwa słowa", 5).is_empty());
assert!(suggest("żółw", 5).len() <= 5);
assert!(!suggest("żółw", 5).contains(&"żółw".to_string()));
}
#[test]
fn distance_weighs_diacritics_half() {
let map = vec![vec!['a', 'ą'], vec!['o', 'ó', 'u']];
let related = |a: char, b: char| fold(a, &map) == fold(b, &map);
let c = |s: &str| s.chars().collect::<Vec<_>>();
assert_eq!(weighted_distance(&c("ktury"), &c("który"), &related, 9), 1);
assert_eq!(weighted_distance(&c("kot"), &c("kto"), &related, 9), 2);
assert_eq!(weighted_distance(&c("kot"), &c("kotek"), &related, 9), 4);
assert_eq!(weighted_distance(&c("kot"), &c("pies"), &related, 2), 3);
assert_eq!(weighted_distance(&c(""), &c("ab"), &related, 9), 4);
}
#[test]
fn suggestions_are_fast_enough() {
crate::pl::test_dict::install();
// Warm the dictionary, then time a worst case (a long word nothing near): the scan round.
let _ = suggest("kot", 1);
let t = std::time::Instant::now();
for w in ["xqzvwpolskiegoslowa", "pszyjacielowi", "zażółćgęśląjaźń"] {
let _ = suggest(w, 6);
}
let ms = t.elapsed().as_secs_f64() * 1000.0;
eprintln!("3 suggestion calls: {ms:.0} ms");
assert!(ms < 20_000.0, "{ms} ms");
}
}
crates/proof/src/pl/test_dict.rs
+108
View File
@@ -0,0 +1,108 @@
//! A tiny synthetic Polish Hunspell dictionary for tests (written for these tests; not taken from
//! any real dictionary). Real dictionaries are installed on the system, never bundled.
/// Affix rules: a few suffixes and one prefix, plus the suggestion tables the Polish
/// suggestions use (`TRY`, `MAP` for diacritics, `REP` for common confusions).
pub const AFF: &str = "SET UTF-8
TRY aioeznrwcysżtkpdmłjlubąęgchśóźćń
MAP 9
MAP aą
MAP cć
MAP eę
MAP lł
MAP nń
MAP oó
MAP uó
MAP sś
MAP zżź
REP 6
REP ż rz
REP rz ż
REP ch h
REP h ch
REP sz rz
REP ść ć
SFX A Y 1
SFX A a ie a
SFX B Y 3
SFX B y ego y
SFX B i iego i
SFX B i iej i
PFX N Y 1
PFX N 0 nie .
";
/// Stems (the count line first, as in Hunspell files).
pub const DIC: &str = "63
źdźbło
gżegżółka
chrząszcz
pięćdziesięciu
rzeka
którzy
żółw
zażółć
gęślą
jaźń
konstytucja
konstytucyjny/B
niebieski/B
niemiecki/B
ładnie/N
przyszliśmy
zrobilibyśmy
Warszawa/A
kot
biało
czerwony
polsko
mail
w
się
huśtawka
brzoza
przyjaciel
wziąć
dziewczyna
na
pewno
ogóle
nie
wiem
płynie
a
i
o
to
że
tak
jest
brzmi
trzcina/A
napisał
poszedł
spać
bo
wylała
świat
dom
ma
ala
kota
dzień
dobry
jutro
bardzo
dobrze
mamy
czas
ten
";
/// Make [`AFF`]/[`DIC`] the Polish dictionary (every test that needs one calls this; they all
/// install the same one, so tests running in parallel agree).
pub fn install() {
if let Err(e) = super::install_dictionary(AFF, DIC) {
panic!("test dictionary: {e}");
}
}
crates/ui-egui/src/canvas.rs
+2 -1
View File
@@ -1984,7 +1984,8 @@ fn context_menu(app: &mut WordApp, ui: &mut Ui) {
.and_then(|v| v.as_array())
.map(|a| a.iter().filter_map(|x| x.as_str().map(str::to_string)).collect())
.unwrap_or_default();
ui.label(egui::RichText::new(issue.get("message").and_then(|m| m.as_str()).unwrap_or("")).small().weak());
// English rules' messages are catalog keys; Polish rules already explain in Polish.
ui.label(egui::RichText::new(tl!(issue.get("message").and_then(|m| m.as_str()).unwrap_or(""))).small().weak());
if sugg.is_empty() {
ui.label(egui::RichText::new(tl!("(no suggestions)")).italics());
}
crates/ui-egui/src/chrome.rs
+7 -1
View File
@@ -291,7 +291,13 @@ pub fn status_bar(app: &mut WordApp, ui: &mut Ui) {
if app.session.sel.focus.story != StoryRef::Body && !crate::canvas::in_text_box(app) {
st(ui, tl!("Editing header/footer"));
}
st(ui, tl!("English (United States)"));
// The language at the caret; a click sets the proofing language (Review › Language).
let lang = crate::ribbon::caret_language(app);
let resp = st(ui, &lang).on_hover_text(tl!("Language"));
egui::Popup::menu(&resp).show(|ui| {
ui.set_min_width(200.0);
crate::ribbon::language_menu(ui, app);
});
if app.session.doc.settings.track_changes {
st(ui, tl!("Track Changes: On"));
}
crates/ui-egui/src/i18n/de.tsv
+1
View File
@@ -2034,3 +2034,4 @@
Your system has animation effects turned on. Auf Ihrem System sind Animationseffekte aktiviert.
Your system has animation effects turned off. Auf Ihrem System sind Animationseffekte deaktiviert.
Your system doesn't report an animation setting, so animation effects stay on. Ihr System meldet keine Animationseinstellung, daher bleiben Animationseffekte aktiviert.
Do Not Check Spelling or Grammar Rechtschreibung und Grammatik nicht prüfen
Can't render this file because it contains an unexpected character in line 6 and column 25.
crates/ui-egui/src/i18n/es.tsv
+3 -2
View File
@@ -1706,6 +1706,9 @@
Continuous numbering Numeración continua
Apply Aplicar
Convert Notes Convertir notas
English (United Kingdom) Inglés (Reino Unido)
Polish Polaco
Do Not Check Spelling or Grammar No revisar la ortografía ni la gramática
# Built-in file browser when the system file dialog is unavailable (#576)
Documents Documentos
@@ -1996,7 +1999,6 @@
Detect language automatically Detectar idioma automáticamente
Do not check spelling or grammar No revisar la ortografía ni la gramática
Dutch Neerlandés
English (United Kingdom) Inglés (Reino Unido)
Envelope size: Tamaño del sobre:
Estonian Estonio
Finnish Finés
@@ -2025,7 +2027,6 @@
Numbering: Numeración:
Omit Omitir
One label per recipient Una etiqueta por destinatario
Polish Polaco
Portuguese (Brazil) Portugués (Brasil)
Portuguese (Portugal) Portugués (Portugal)
Return address labels Etiquetas de remitente
Can't render this file because it contains an unexpected character in line 6 and column 25.
crates/ui-egui/src/i18n/et.tsv
+3 -2
View File
@@ -1667,6 +1667,9 @@
Continuous numbering Pidev nummerdus
Apply Rakenda
Convert Notes Teisenda märkused
Do Not Check Spelling or Grammar Ära kontrolli õigekirja ega grammatikat
English (United Kingdom) Inglise (Ühendkuningriik)
Polish Poola
# Built-in file browser when the system file dialog is unavailable (#576)
Documents Dokumendid
@@ -1990,7 +1993,6 @@
Detect language automatically Tuvasta keel automaatselt
Do not check spelling or grammar Ära kontrolli õigekirja ega grammatikat
Dutch Hollandi
English (United Kingdom) Inglise (Ühendkuningriik)
Envelope size: Ümbriku suurus:
Estonian Eesti
Finnish Soome
@@ -2019,7 +2021,6 @@
Numbering: Nummerdus:
Omit Jäta välja
One label per recipient Üks silt saaja kohta
Polish Poola
Portuguese (Brazil) Portugali (Brasiilia)
Portuguese (Portugal) Portugali (Portugal)
Return address labels Saatja aadressi sildid
Can't render this file because it contains an unexpected character in line 445 and column 13.
crates/ui-egui/src/i18n/it.tsv
+1
View File
@@ -2419,3 +2419,4 @@
{name} wasn't found. {name} non è stato trovato.
✓ None ✓ Nessuno
preview Title Esempio
Do Not Check Spelling or Grammar Non controllare ortografia e grammatica
Can't render this file because it contains an unexpected character in line 6 and column 25.
crates/ui-egui/src/i18n/ja.tsv
+3 -2
View File
@@ -1690,6 +1690,9 @@
Continuous numbering 通し番号
Apply 適用
Convert Notes 注の変換
English (United Kingdom) 英語（英国）
Polish ポーランド語
Do Not Check Spelling or Grammar スペル チェックと文章校正を行わない
# Built-in file browser when the system file dialog is unavailable (#576)
Documents ドキュメント
@@ -1980,7 +1983,6 @@
Detect language automatically 言語を自動的に検出する
Do not check spelling or grammar スペルチェックと文章校正を行わない
Dutch オランダ語
English (United Kingdom) 英語（英国）
Envelope size: 封筒サイズ:
Estonian エストニア語
Finnish フィンランド語
@@ -2009,7 +2011,6 @@
Numbering: 番号の付け方:
Omit 省略
One label per recipient 宛先ごとに 1 枚のラベル
Polish ポーランド語
Portuguese (Brazil) ポルトガル語（ブラジル）
Portuguese (Portugal) ポルトガル語（ポルトガル）
Return address labels 差出人ラベル
Can't render this file because it contains an unexpected character in line 553 and column 13.
crates/ui-egui/src/i18n/mod.rs
+65 -16
View File
@@ -14,7 +14,8 @@
//! # Adding a language
//! 1. Add `xx.tsv` next to `zh-hans.tsv` (copy its header; translate from the *meaning* of the
//! English text, clean-room: never from Microsoft Word's or another product's localisation).
//! 2. Add one row to [`LANGUAGES`].
//! 2. Add one row to [`LANGUAGES`]. Set its `editing_lang` only when `wordcraft-proof` can proof
//! the language (Polish: `pl-PL`): new documents are then written in it.
//!
//! The Options dropdown, the system-language match and the catalog tests pick it up from there.
//!
@@ -44,38 +45,81 @@ pub struct LangInfo {
pub source: &'static str,
/// Which CJK interface face comes first (Chinese text wants the Chinese face).
pub prefer_hans: bool,
/// The language (BCP 47) new blank documents are written in with this interface, for
/// languages WordCraft can proof; `None` keeps WordCraft's default, US English.
pub editing_lang: Option<&'static str>,
catalog: OnceLock<Catalog>,
}
/// The registry. English first: it is the fallback and the source language.
pub static LANGUAGES: [LangInfo; 14] = [
LangInfo { code: "en", name: "English", source: "", prefer_hans: false, catalog: OnceLock::new() },
pub static LANGUAGES: [LangInfo; 15] = [
LangInfo { code: "en", name: "English", source: "", prefer_hans: false, editing_lang: None, catalog: OnceLock::new() },
// Simplified Chinese; `zh`, `zh-CN`, `zh-SG` and `zh-Hans-*` resolve here (see `candidates`).
LangInfo { code: "zh-hans", name: "简体中文", source: include_str!("zh-hans.tsv"), prefer_hans: true, catalog: OnceLock::new() },
LangInfo {
code: "zh-hans",
name: "简体中文",
source: include_str!("zh-hans.tsv"),
prefer_hans: true,
editing_lang: None,
catalog: OnceLock::new(),
},
// Traditional Chinese (Taiwan vocabulary); `zh-TW`, `zh-HK`, `zh-MO` and `zh-Hant-*` resolve here.
LangInfo { code: "zh-hant", name: "繁體中文", source: include_str!("zh-hant.tsv"), prefer_hans: true, catalog: OnceLock::new() },
LangInfo { code: "ja", name: "日本語", source: include_str!("ja.tsv"), prefer_hans: false, catalog: OnceLock::new() },
LangInfo { code: "uk", name: "Українська", source: include_str!("uk.tsv"), prefer_hans: false, catalog: OnceLock::new() },
LangInfo {
code: "zh-hant",
name: "繁體中文",
source: include_str!("zh-hant.tsv"),
prefer_hans: true,
editing_lang: None,
catalog: OnceLock::new(),
},
LangInfo { code: "ja", name: "日本語", source: include_str!("ja.tsv"), prefer_hans: false, editing_lang: None, catalog: OnceLock::new() },
LangInfo {
code: "uk", name: "Українська", source: include_str!("uk.tsv"), prefer_hans: false, editing_lang: None, catalog: OnceLock::new()
},
// Spanish, neutral across Spain and Latin America; `es-ES`, `es-MX`, `es-419` … resolve here.
LangInfo { code: "es", name: "Español", source: include_str!("es.tsv"), prefer_hans: false, catalog: OnceLock::new() },
LangInfo { code: "es", name: "Español", source: include_str!("es.tsv"), prefer_hans: false, editing_lang: None, catalog: OnceLock::new() },
// Brazilian Portuguese; `pt-BR` and `pt-BR-*` resolve here. Plain `pt` and `pt-PT` have no
// catalog yet (the European vocabulary differs), so they stay in English.
LangInfo { code: "pt-br", name: "Português (Brasil)", source: include_str!("pt-br.tsv"), prefer_hans: false, catalog: OnceLock::new() },
LangInfo {
code: "pt-br",
name: "Português (Brasil)",
source: include_str!("pt-br.tsv"),
prefer_hans: false,
editing_lang: None,
catalog: OnceLock::new(),
},
// Serbian, Cyrillic script (the default per BCP 47 when no script is given); `sr`, `sr-RS`,
// `sr-Cyrl-*` resolve here.
LangInfo { code: "sr", name: "Српски", source: include_str!("sr.tsv"), prefer_hans: false, catalog: OnceLock::new() },
LangInfo { code: "sr", name: "Српски", source: include_str!("sr.tsv"), prefer_hans: false, editing_lang: None, catalog: OnceLock::new() },
// Serbian, Latin script; `sr-Latn-*` resolves here (see `candidates`'s generic prefix
// matching — no special-casing needed, unlike Chinese's script-by-region fallback).
LangInfo { code: "sr-latn", name: "Srpski (latinica)", source: include_str!("sr-latn.tsv"), prefer_hans: false, catalog: OnceLock::new() },
LangInfo { code: "et", name: "Eesti", source: include_str!("et.tsv"), prefer_hans: false, catalog: OnceLock::new() },
LangInfo {
code: "sr-latn",
name: "Srpski (latinica)",
source: include_str!("sr-latn.tsv"),
prefer_hans: false,
editing_lang: None,
catalog: OnceLock::new(),
},
// Polish; `pl`, `pl-PL` and `pl_PL.UTF-8` resolve here. WordCraft proofs Polish (spelling,
// grammar, hyphenation), so new documents are written in Polish with this interface.
LangInfo {
code: "pl",
name: "Polski",
source: include_str!("pl.tsv"),
prefer_hans: false,
editing_lang: Some("pl-PL"),
catalog: OnceLock::new(),
},
LangInfo { code: "et", name: "Eesti", source: include_str!("et.tsv"), prefer_hans: false, editing_lang: None, catalog: OnceLock::new() },
// German, neutral across Germany, Austria and Switzerland; `de`, `de-DE`, `de-AT`, `de-CH` …
// resolve here.
LangInfo { code: "de", name: "Deutsch", source: include_str!("de.tsv"), prefer_hans: false, catalog: OnceLock::new() },
LangInfo { code: "de", name: "Deutsch", source: include_str!("de.tsv"), prefer_hans: false, editing_lang: None, catalog: OnceLock::new() },
// Russian; `ru`, `ru-RU`, `ru_RU.UTF-8` and other regions resolve here.
LangInfo { code: "ru", name: "Русский", source: include_str!("ru.tsv"), prefer_hans: false, catalog: OnceLock::new() },
LangInfo { code: "tr", name: "Türkçe", source: include_str!("tr.tsv"), prefer_hans: false, catalog: OnceLock::new() },
LangInfo { code: "ru", name: "Русский", source: include_str!("ru.tsv"), prefer_hans: false, editing_lang: None, catalog: OnceLock::new() },
LangInfo { code: "tr", name: "Türkçe", source: include_str!("tr.tsv"), prefer_hans: false, editing_lang: None, catalog: OnceLock::new() },
// Italian; `it-IT`, `it-CH` and `it-*` resolve here.
LangInfo { code: "it", name: "Italiano", source: include_str!("it.tsv"), prefer_hans: false, catalog: OnceLock::new() },
LangInfo { code: "it", name: "Italiano", source: include_str!("it.tsv"), prefer_hans: false, editing_lang: None, catalog: OnceLock::new() },
];
impl LangInfo {
@@ -123,6 +167,11 @@ impl Lang {
self.0.prefer_hans
}
/// The language new blank documents are written in with this interface ([`LangInfo::editing_lang`]).
pub fn editing_lang(self) -> Option<&'static str> {
self.0.editing_lang
}
/// Is the interface written in CJK script (its own name is), so it needs a CJK face?
pub fn uses_cjk(self) -> bool {
self.0.name.chars().any(|c| ('\u{2E80}'..='\u{9FFF}').contains(&c) || ('\u{AC00}'..='\u{D7AF}').contains(&c))
crates/ui-egui/src/i18n/pl.tsv
+2055
View File
File diff suppressed because it is too large. Load diff
crates/ui-egui/src/i18n/pt-br.tsv
+3 -2
View File
@@ -1702,6 +1702,9 @@
Continuous numbering Numeração contínua
Apply Aplicar
Convert Notes Converter notas
English (United Kingdom) Inglês (Reino Unido)
Polish Polonês
Do Not Check Spelling or Grammar Não verificar ortografia nem gramática
# Built-in file browser when the system file dialog is unavailable (#576)
Documents Documentos
@@ -1992,7 +1995,6 @@
Detect language automatically Detectar o idioma automaticamente
Do not check spelling or grammar Não verificar ortografia nem gramática
Dutch Holandês
English (United Kingdom) Inglês (Reino Unido)
Envelope size: Tamanho do envelope:
Estonian Estoniano
Finnish Finlandês
@@ -2021,7 +2023,6 @@
Numbering: Numeração:
Omit Omitir
One label per recipient Uma etiqueta por destinatário
Polish Polonês
Portuguese (Brazil) Português (Brasil)
Portuguese (Portugal) Português (Portugal)
Return address labels Etiquetas de remetente
Can't render this file because it contains an unexpected character in line 6 and column 25.
crates/ui-egui/src/i18n/ru.tsv
+3 -2
View File
@@ -2135,6 +2135,9 @@ preview Title Пример
Continuous numbering Сквозная нумерация
Apply Применить
Convert Notes Преобразовать сноски
Do Not Check Spelling or Grammar Не проверять орфографию и грамматику
English (United Kingdom) Английский (Великобритания)
Polish Польский
# Built-in file browser (#576).
Documents Документы
@@ -2326,7 +2329,6 @@ preview Title Пример
Detect language automatically Определять язык автоматически
Do not check spelling or grammar Не проверять орфографию и грамматику
Dutch Нидерландский
English (United Kingdom) Английский (Великобритания)
Envelope size: Размер конверта:
Estonian Эстонский
Finnish Финский
@@ -2355,7 +2357,6 @@ preview Title Пример
Numbering: Нумерация:
Omit Пропустить
One label per recipient Одна наклейка на получателя
Polish Польский
Portuguese (Brazil) Португальский (Бразилия)
Portuguese (Portugal) Португальский (Португалия)
Return address labels Наклейки с обратным адресом
Can't render this file because it contains an unexpected character in line 7 and column 25.
crates/ui-egui/src/i18n/sr-latn.tsv
+3 -2
View File
@@ -1700,6 +1700,9 @@
Continuous numbering Neprekidna numeracija
Apply Primeni
Convert Notes Pretvori napomene
English (United Kingdom) Engleski (Velika Britanija)
Polish Poljski
Do Not Check Spelling or Grammar Ne proveravaj pravopis i gramatiku
# Built-in file browser when the system file dialog is unavailable (#576)
Documents Dokumenti
@@ -1990,7 +1993,6 @@
Detect language automatically Automatski prepoznaj jezik
Do not check spelling or grammar Ne proveravaj pravopis ni gramatiku
Dutch Holandski
English (United Kingdom) Engleski (Ujedinjeno Kraljevstvo)
Envelope size: Veličina koverte:
Estonian Estonski
Finnish Finski
@@ -2019,7 +2021,6 @@
Numbering: Numerisanje:
Omit Izostavi
One label per recipient Jedna nalepnica po primaocu
Polish Poljski
Portuguese (Brazil) Portugalski (Brazil)
Portuguese (Portugal) Portugalski (Portugalija)
Return address labels Nalepnice sa adresom pošiljaoca
Can't render this file because it contains an unexpected character in line 6 and column 25.
crates/ui-egui/src/i18n/sr.tsv
+3 -2
View File
@@ -1704,6 +1704,9 @@
Continuous numbering Непрекидна нумерација
Apply Примени
Convert Notes Претвори напомене
English (United Kingdom) Енглески (Велика Британија)
Polish Пољски
Do Not Check Spelling or Grammar Не проверавај правопис и граматику
# Built-in file browser when the system file dialog is unavailable (#576)
Documents Документи
@@ -1994,7 +1997,6 @@
Detect language automatically Аутоматски препознај језик
Do not check spelling or grammar Не проверавај правопис ни граматику
Dutch Холандски
English (United Kingdom) Енглески (Уједињено Краљевство)
Envelope size: Величина коверте:
Estonian Естонски
Finnish Фински
@@ -2023,7 +2025,6 @@
Numbering: Нумерисање:
Omit Изостави
One label per recipient Једна налепница по примаоцу
Polish Пољски
Portuguese (Brazil) Португалски (Бразил)
Portuguese (Portugal) Португалски (Португалија)
Return address labels Налепнице са адресом пошиљаоца
Can't render this file because it contains an unexpected character in line 11 and column 25.
crates/ui-egui/src/i18n/tests.rs
+107
View File
@@ -653,6 +653,113 @@ fn bundled_interface_fonts_cover_turkish_without_system_fallbacks() {
assert!(!lang("tr").uses_cjk());
}
#[test]
fn polish_locales_and_saved_preference_work_without_changing_the_document() {
let pl = lang("pl");
for tag in ["pl", "pl-PL", "pl_PL.UTF-8", "PL-pl", "pl_PL.UTF-8@euro", "pl-Latn-PL"] {
assert_eq!(lang_from_tag(tag), Some(pl), "{tag}");
}
assert_eq!(first_supported(["fr-FR", "pl-PL", "en-US"]), Some(pl));
let mut app = crate::WordApp::new(wordcraft_engine::Session::new(wordcraft_engine::sample::sample_document()), Default::default());
let before = serde_json::to_value(&app.session.doc).unwrap();
let result = app.run("ui.language", serde_json::json!({"value": "PL"})).unwrap();
assert_eq!(result["effective"], "pl");
assert_eq!(app.ui.language, "pl");
assert_eq!(serde_json::to_value(&app.session.doc).unwrap(), before);
let saved = serde_json::to_string(&app.ui).unwrap();
let restored: crate::UiState = serde_json::from_str(&saved).unwrap();
assert_eq!(Lang::from_pref(&restored.language), pl);
assert_eq!(pl.name(), "Polski");
}
#[test]
fn polish_covers_every_key_of_every_other_catalog() {
use std::collections::HashSet;
let keys = |source| parse_entries(source).0.into_iter().map(|e| e.source).collect::<HashSet<_>>();
let polish = keys(lang("pl").0.source);
// Russian and Italian also translate the built-in style names and the equation gallery,
// which the others don't yet; Polish matches every other catalog.
for other in Lang::all().filter(|l| *l != Lang::EN && !matches!(l.code(), "pl" | "ru" | "it")) {
let mut missing: Vec<String> = keys(other.0.source).difference(&polish).cloned().collect();
missing.sort();
assert!(missing.is_empty(), "pl lacks {}'s {missing:?}", other.code());
}
let pl = lang("pl");
assert_eq!(tr(pl, "File"), "Plik");
assert_eq!(tr(pl, "Home"), "Start");
assert_eq!(tr(pl, "Insert"), "Wstaw");
assert_eq!(tr(pl, "Review"), "Recenzja");
assert_eq!(tr(pl, "Save"), "Zapisz");
assert_eq!(tr(pl, "Spelling & Grammar"), "Pisownia i gramatyka");
assert_eq!(tr(pl, "Polish"), "Polski");
assert_eq!(tr(pl, "unknown future label"), "unknown future label");
assert_eq!(fmt(tr(pl, "Exported {path}"), &[("path", "draft-{words}.docx")]), "Wyeksportowano: draft-{words}.docx");
assert_eq!(fmt(tr(pl, "Page {page} of {pages}"), &[("page", "3"), ("pages", "9")]), "Strona 3 z 9");
// Count-neutral wording reads grammatically for every Polish number form (1, 2–4, 5+, 12–14, 22).
for count in [0, 1, 2, 4, 5, 12, 14, 22, 25, 101, 1000] {
let words = count.to_string();
assert_eq!(fmt(tr(pl, "{words} words"), &[("words", &words)]), format!("Słowa: {count}"));
assert_eq!(fmt(tr(pl, "{selected} of {words} words"), &[("selected", "1"), ("words", &words)]), format!("Słowa: 1 z {count}"));
}
set_current(pl);
assert_eq!(location("Review › Proofing"), "Recenzja › Sprawdzanie");
assert_eq!(prefixed("Undo", "Bold"), "Cofnij: Pogrubienie");
set_current(Lang::EN);
}
#[test]
fn bundled_interface_fonts_cover_polish_without_system_fallbacks() {
let db = wordcraft_fonts::FontDb::with_font_dirs(Vec::new());
let (entries, errors) = parse_entries(lang("pl").0.source);
assert!(errors.is_empty(), "{errors:?}");
// Polish letters beyond ASCII, both cases, and the Polish quotation marks.
let polish = "ĄąĆćĘꣳŃńÓ󌜏źŻż„”";
let chars: std::collections::HashSet<char> =
entries.iter().flat_map(|e| e.translation.chars()).filter(|c| polish.contains(*c)).chain(polish.chars()).collect();
for (family, style) in [("Inter", "Regular"), ("Inter", "Medium"), ("Inter", "SemiBold"), ("JetBrains Mono", "Regular")] {
let face = db.face(family, style);
assert_eq!(face.family, family, "must use the bundled interface face");
for ch in &chars {
assert_ne!(face.glyph_for(*ch), 0, "{family} {style} lacks {ch}");
}
}
}
#[test]
fn a_polish_interface_writes_new_documents_in_polish() {
let mut app = crate::WordApp::new(wordcraft_engine::Session::new(wordcraft_doc::Document::new()), Default::default());
let ctx = egui::Context::default();
let frame = |app: &mut crate::WordApp| ctx.run_ui(egui::RawInput::default(), |ui| app.logic(ui.ctx())).drop_without_applying_deltas();
frame(&mut app);
assert_eq!(app.session.editing_lang, None, "English interface: WordCraft's default, US English");
assert_eq!(app.session.doc.styles.default_chr.lang.as_deref(), Some("en-US"));
app.run("ui.language", serde_json::json!({"value": "pl"})).unwrap();
frame(&mut app);
assert_eq!(app.session.editing_lang.as_deref(), Some("pl-PL"));
assert_eq!(app.session.doc.styles.default_chr.lang.as_deref(), Some("pl-PL"), "the untouched first document follows");
assert!(!app.session.dirty);
app.run("file.new", serde_json::json!({})).unwrap();
assert_eq!(app.session.doc.styles.default_chr.lang.as_deref(), Some("pl-PL"));
// Typing Polish then shows no spelling marks; the status bar names the language.
app.session.run("text.insert", &serde_json::json!({"text": "Zażółć gęślą jaźń"})).unwrap();
assert_eq!(app.session.run("review.issues", &serde_json::json!({})).unwrap(), serde_json::json!([]));
set_current(lang("pl"));
assert_eq!(crate::ribbon::caret_language(&app), "Polski");
set_current(Lang::EN);
assert_eq!(crate::ribbon::caret_language(&app), "Polish");
// Back to English: documents already written keep their language.
app.run("ui.language", serde_json::json!({"value": "en"})).unwrap();
frame(&mut app);
assert_eq!(app.session.editing_lang, None);
assert_eq!(app.session.doc.styles.default_chr.lang.as_deref(), Some("pl-PL"));
}
#[test]
fn only_polish_has_an_editing_language() {
let with: Vec<(&str, &str)> = Lang::all().filter_map(|l| l.editing_lang().map(|e| (l.code(), e))).collect();
assert_eq!(with, [("pl", "pl-PL")]);
}
#[test]
fn german_locales_and_saved_preference_keep_document_content() {
let de = lang("de");
crates/ui-egui/src/i18n/tr.tsv
+3 -2
View File
@@ -1331,6 +1331,9 @@
Edit Field Code Alan Kodunu Düzenle
Edit Field… Alanı Düzenle…
Go Back Geri Git
Do Not Check Spelling or Grammar Yazım veya dilbilgisini denetleme
English (United Kingdom) İngilizce (Birleşik Krallık)
Polish Lehçe
Insert Date Field Tarih Alanı Ekle
Insert Page Field Sayfa Alanı Ekle
Insert Time Field Saat Alanı Ekle
@@ -1948,7 +1951,6 @@
Detect language automatically Dili otomatik algıla
Do not check spelling or grammar Yazım ve dil bilgisini denetleme
Dutch Felemenkçe
English (United Kingdom) İngilizce (Birleşik Krallık)
Envelope size: Zarf boyutu:
Estonian Estonca
Finnish Fince
@@ -1977,7 +1979,6 @@
Numbering: Numaralandırma:
Omit Atla
One label per recipient Alıcı başına bir etiket
Polish Lehçe
Portuguese (Brazil) Portekizce (Brezilya)
Portuguese (Portugal) Portekizce (Portekiz)
Return address labels Gönderen adresi etiketleri
Can't render this file because it contains an unexpected character in line 431 and column 13.
crates/ui-egui/src/i18n/uk.tsv
+3 -2
View File
@@ -1702,6 +1702,9 @@
Continuous numbering Наскрізна нумерація
Apply Застосувати
Convert Notes Перетворити примітки
English (United Kingdom) Англійська (Велика Британія)
Polish Польська
Do Not Check Spelling or Grammar Не перевіряти правопис і граматику
# Built-in file browser when the system file dialog is unavailable (#576)
Documents Документи
@@ -1992,7 +1995,6 @@
Detect language automatically Визначати мову автоматично
Do not check spelling or grammar Не перевіряти правопис і граматику
Dutch Нідерландська
English (United Kingdom) Англійська (Велика Британія)
Envelope size: Розмір конверта:
Estonian Естонська
Finnish Фінська
@@ -2021,7 +2023,6 @@
Numbering: Нумерація:
Omit Пропустити
One label per recipient Одна етикетка на одержувача
Polish Польська
Portuguese (Brazil) Португальська (Бразилія)
Portuguese (Portugal) Португальська (Португалія)
Return address labels Етикетки зі зворотною адресою
Can't render this file because it contains an unexpected character in line 6 and column 31.
crates/ui-egui/src/i18n/zh-hans.tsv
+3 -2
View File
@@ -1702,6 +1702,9 @@
Continuous numbering 连续编号
Apply 应用
Convert Notes 转换注释
English (United Kingdom) 英语（英国）
Polish 波兰语
Do Not Check Spelling or Grammar 不检查拼写和语法
# Built-in file browser when the system file dialog is unavailable (#576)
Documents 文档
@@ -1992,7 +1995,6 @@
Detect language automatically 自动检测语言
Do not check spelling or grammar 不检查拼写或语法
Dutch 荷兰语
English (United Kingdom) 英语（英国）
Envelope size: 信封尺寸：
Estonian 爱沙尼亚语
Finnish 芬兰语
@@ -2021,7 +2023,6 @@
Numbering: 编号：
Omit 省略
One label per recipient 每位收件人一个标签
Polish 波兰语
Portuguese (Brazil) 葡萄牙语（巴西）
Portuguese (Portugal) 葡萄牙语（葡萄牙）
Return address labels 回信地址标签
Can't render this file because it contains an unexpected character in line 7 and column 25.
crates/ui-egui/src/i18n/zh-hant.tsv
+3 -2
View File
@@ -1691,6 +1691,9 @@
Continuous numbering 連續編號
Apply 套用
Convert Notes 轉換附註
English (United Kingdom) 英文（英國）
Polish 波蘭文
Do Not Check Spelling or Grammar 不檢查拼字及文法
# Built-in file browser when the system file dialog is unavailable (#576)
Documents 文件
@@ -1981,7 +1984,6 @@
Detect language automatically 自動偵測語言
Do not check spelling or grammar 不要檢查拼字或文法
Dutch 荷蘭文
English (United Kingdom) 英文（英國）
Envelope size: 信封大小：
Estonian 愛沙尼亞文
Finnish 芬蘭文
@@ -2010,7 +2012,6 @@
Numbering: 編號：
Omit 省略
One label per recipient 每位收件人一個標籤
Polish 波蘭文
Portuguese (Brazil) 葡萄牙文（巴西）
Portuguese (Portugal) 葡萄牙文（葡萄牙）
Return address labels 寄件人地址標籤
Can't render this file because it contains an unexpected character in line 554 and column 13.
crates/ui-egui/src/lib.rs
+11
View File
@@ -313,6 +313,8 @@ pub struct WordApp {
fonts_frames: u32,
/// The interface theme setting last installed in egui ([`theme::apply`]).
applied_theme: Option<theme::Appearance>,
/// The interface language whose editing language the session last got ([`i18n::Lang::editing_lang`]).
applied_editing: Option<i18n::Lang>,
/// What the system last said about its animation effects ([`Services::animations_enabled`]).
system_animations: Option<bool>,
/// Whether egui's own animations were last switched on or off ([`theme::apply_animations`]).
@@ -399,6 +401,7 @@ impl WordApp {
want_system_cjk: false,
fonts_frames: 0,
applied_theme: None,
applied_editing: None,
system_animations: None,
applied_animations: None,
frame_ms: 0.0,
@@ -763,7 +766,9 @@ impl WordApp {
self.ui.recent.truncate(MAX_RECENT);
}
if !self.session.status.is_empty() {
// Engine messages are English; the catalog translates the ones it knows.
let s = std::mem::take(&mut self.session.status);
let s = tl!(&s).to_string();
self.status(s);
}
}
@@ -1117,6 +1122,12 @@ impl WordApp {
pub fn logic(&mut self, ctx: &egui::Context) {
let lang = i18n::Lang::from_pref(&self.ui.language);
i18n::set_current(lang);
// New documents are written in the interface's language when WordCraft can proof it
// (a Polish interface: Polish spelling, grammar and hyphenation from the first word).
if self.applied_editing != Some(lang) {
self.session.set_editing_language(lang.editing_lang());
self.applied_editing = Some(lang);
}
options::sync(self);
// Chinese text wants the Chinese face before the Japanese one (one glyph style per line).
// An installed CJK font is read only once CJK text is shown (#241): never for an English
crates/ui-egui/src/ribbon.rs
+41 -1
View File
@@ -342,6 +342,46 @@ fn mi_text(ui: &mut Ui, app: &mut WordApp, text: &str, id: &str, params: Value)
}
}
/// The proofing languages the Language menu offers: tag and English name (translated when drawn).
pub(crate) const PROOFING_LANGUAGES: [(&str, &str); 3] =
[("en-US", "English (United States)"), ("en-GB", "English (United Kingdom)"), ("pl-PL", "Polish")];
/// The English name of a proofing language tag, as the status bar shows it (no tag: WordCraft's
/// default, US English; a tag without a name here: the tag).
pub(crate) fn language_name(tag: Option<&str>) -> String {
let tag = tag.unwrap_or("en-US");
if let Some((_, name)) = PROOFING_LANGUAGES.iter().find(|(t, _)| t.eq_ignore_ascii_case(tag)) {
return (*name).to_string();
}
match wordcraft_engine::proof::ProofLang::from_tag(Some(tag)) {
wordcraft_engine::proof::ProofLang::Pl => "Polish".into(),
wordcraft_engine::proof::ProofLang::En if tag.eq_ignore_ascii_case("en") => "English (United States)".into(),
_ => tag.to_string(),
}
}
/// The language at the caret (resolved through the styles), translated, for the status bar.
pub(crate) fn caret_language(app: &WordApp) -> String {
let s = &app.session;
let style = s.doc.para_at(&s.sel.focus).and_then(|p| p.props.style.clone());
let rc = s.doc.styles.resolve_char(style.as_deref(), &s.typing_props());
tl!(&language_name(rc.lang.as_deref())).to_string()
}
/// Review › Language (and the status bar's language): set the selection's proofing language, or
/// leave it unchecked. The current choice has a check mark.
pub(crate) fn language_menu(ui: &mut Ui, app: &mut WordApp) {
let st = app.session.run("format.state", &json!({})).unwrap_or_default();
let current = st.get("lang").and_then(Value::as_str).unwrap_or("").to_string();
let no_proof = st.get("noProof").and_then(Value::as_bool).unwrap_or(false);
for (tag, name) in PROOFING_LANGUAGES {
let on = current.eq_ignore_ascii_case(tag) || (tag == "pl-PL" && current.eq_ignore_ascii_case("pl"));
mi_check(ui, app, name, on, "review.language", json!({"lang": tag}));
}
ui.separator();
mi_check(ui, app, "Do Not Check Spelling or Grammar", no_proof, "review.language", json!({"noProof": !no_proof}));
}
/// A menu item with a check mark when `checked` (the current choice of several).
fn mi_check(ui: &mut Ui, app: &mut WordApp, label: &str, checked: bool, id: &str, params: Value) {
// An invisible mark keeps the unchecked labels aligned with the checked one.
@@ -1387,7 +1427,7 @@ fn review(app: &mut WordApp, ui: &mut Ui) {
});
group(ui, "Language", None, app, |ui, app| {
big(ui, app, "translate", "Translate", "review.translate", json!({}), false);
big(ui, app, "language", "Language", "review.language", json!({}), false);
menu_button(ui, app, "language", Some("Language"), "Language", true, language_menu);
});
group(ui, "Comments", None, app, |ui, app| {
big(ui, app, "newComment", "New\nComment", "review.newComment", json!({}), false);
docs/architecture.md
+3 -2
View File
@@ -1,6 +1,6 @@
# WordCraft architecture
> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-10 · **Change:** major (first committed version; `plan/architecture.md` is local-only) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4
> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-10 · **Change:** minor (`proof` is per language: English and Polish; previously major: first committed version; `plan/architecture.md` is local-only) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4
How WordCraft is built today (origin/main `40c5d51`). The longer design notes in
`plan/architecture.md` are gitignored and local to the original author's machine; this file is the
@@ -17,7 +17,7 @@ winit or rfd.
| L0 | `geom` | 0.2k | Units (points, twips, EMU, inches, cm), rectangles, measurement parsing |
| L1 | `doc` | 10.0k | Document model: stories, paragraphs, runs, tables, sections, styles, numbering, fields, notes, comments, revisions, equations (OMML-shaped math tree, linear/LaTeX input); edit primitives; bidi |
| L1 | `fonts` | 2.3k | Font database (bundled OFL + system fonts), metrics, outlines, shaping with HarfRust, Word font substitution (`word.rs`: Carlito ↔ Calibri, Caladea ↔ Cambria, Liberation ↔ Arial/Times/Courier), OpenType MATH |
| L1 | `proof` | 1.4k | Spelling (public-domain English word list), suggestions, rule-based grammar, hyphenation patterns |
| L1 | `proof` | 3.8k | Proofing per language (`ProofLang` from a run's `w:lang`): English (public-domain word list) and Polish (installed Hunspell dictionary through a Hunspell-style affix engine) spelling and suggestions, rule-based grammar for both, hyphenation dictionary and Liang patterns (English, Polish) |
| L2 | `layout` | 9.6k | Shaping runs, line breaking, tabs, lists, tables (row splitting, floating tables), pagination, headers/footers, notes, floating objects and wrap, math layout, fields, hit testing; produces a display list |
| L2 | `docx` | 8.4k | OOXML WordprocessingML reader and writer (Transitional and Strict namespaces, `.docx/.docm/.dotx/.dotm`), OMML, custom properties, VBA parts preserved |
| L2 | `docbin` | 4.4k | Word 97-2003 `.doc` reader ([MS-DOC]: FIB, piece table, FKPs, SPRMs, stylesheet, lists, tables, notes, pictures) over a CFB container |
@@ -64,4 +64,5 @@ winit or rfd.
| Date | Change | Summary |
|---|---|---|
| 2026-10-10 | minor | `proof`: proofing languages (English, Polish), affix engine, Polish patterns; layout and engine route each run by its language |
| 2026-10-10 | major | First committed architecture doc, from the crates' sources and `AGENTS.md` |
docs/control-protocol.md
+1 -1
View File
@@ -50,7 +50,7 @@ and set `WORDCRAFT_CONTROL_KEY` to that app's key.
| Method | Params | Result |
|---|---|---|
| `engine.execute` | `command`, `params` | the command's result (any id from `engine.commands`, including UI commands `ui.tab`, `ui.dialog`, `ui.backstage`, `ui.zotero.*` (see `docs/zotero.md`), `ui.language` (`{"value": "auto"|"en"|"zh-hans"|"zh-hant"|"ja"|"pt-br"|"es"|"uk"|"sr"|"sr-latn"|"et"|"tr"|"ru"|"de"|"it"}`), `ui.theme` (`{"value": "system"|"light"|"dark"}`; `system` follows the OS appearance, light when it reports none), `ui.interfaceSize` (`{"percent": 80–200}`, clamped; no value reads it; scales the interface on top of the system display scale, document zoom stays `view.zoom`), `ui.animations` (`{"value": "system"|"on"|"off"}`; no value reads it; `system` follows the system's animation setting, which the desktop app reads on Windows, macOS, Linux and the BSDs and the web app from `prefers-reduced-motion`, and means on where the system doesn't say; the result's `enabled` is what the setting comes to and `system` what the system reports, `null` when it doesn't say), `ui.fileBrowser` (desktop: `{"purpose": "document"|"picture"|"recipients"|"save"}` opens WordCraft's own file browser, for when the system file dialog doesn't show up, #576), `ui.caretAnimation` (`{"value": true|false}`; no value reads it; off, the caret jumps and blinks on and off instead of gliding and fading)…) |
| `engine.execute` | `command`, `params` | the command's result (any id from `engine.commands`, including UI commands `ui.tab`, `ui.dialog`, `ui.backstage`, `ui.zotero.*` (see `docs/zotero.md`), `ui.language` (`{"value": "auto"|"en"|"zh-hans"|"zh-hant"|"ja"|"pt-br"|"es"|"uk"|"sr"|"sr-latn"|"et"|"tr"|"ru"|"pl"|"de"|"it"}`), `ui.theme` (`{"value": "system"|"light"|"dark"}`; `system` follows the OS appearance, light when it reports none), `ui.interfaceSize` (`{"percent": 80–200}`, clamped; no value reads it; scales the interface on top of the system display scale, document zoom stays `view.zoom`), `ui.animations` (`{"value": "system"|"on"|"off"}`; no value reads it; `system` follows the system's animation setting, which the desktop app reads on Windows, macOS, Linux and the BSDs and the web app from `prefers-reduced-motion`, and means on where the system doesn't say; the result's `enabled` is what the setting comes to and `system` what the system reports, `null` when it doesn't say), `ui.fileBrowser` (desktop: `{"purpose": "document"|"picture"|"recipients"|"save"}` opens WordCraft's own file browser, for when the system file dialog doesn't show up, #576), `ui.caretAnimation` (`{"value": true|false}`; no value reads it; off, the caret jumps and blinks on and off instead of gliding and fading)…) |
| `engine.commands` | — | every command: id, label, location, shortcut, params, enabled |
| `document.inspect` | `text?` | blocks (text, style, runs, lists, tables), parts, sections, selection, pages |
| `ui.inspect` | — | UI state, view, dialog, window size, page rects on screen, caret, comment balloons (`balloons`: id + screen rect; `balloon`: the selected one), perf |
docs/gaps.md
+3 -2
View File
@@ -1,6 +1,6 @@
# Where WordCraft falls short of Word
> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-11 · **Change:** trivial (.doc pictures and drawings read, #551; previously trivial: #24 narrowed: Customize Keyboard and Customize Ribbon landed; previously trivial: formatting revisions landed, #41; previously trivial: Draw Table and Eraser landed, #303; previously minor: beta distance restated: ~15 points, ~150–230 h; previously: first version; every known shortfall from the 2026-10-10 re-measure, ranked) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4
> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-11 · **Change:** minor (Polish interface and Polish proofing; previously trivial: .doc pictures and drawings read, #551; previously trivial: #24 narrowed: Customize Keyboard and Customize Ribbon landed; previously trivial: formatting revisions landed, #41; previously trivial: Draw Table and Eraser landed, #303; previously minor: beta distance restated: ~15 points, ~150–230 h; previously: first version; every known shortfall from the 2026-10-10 re-measure, ranked) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4
The work list. One entry per shortfall, ranked by how much it stops a Word user from switching
(**B** = blocks beta). Each says what's missing, the evidence, who it hurts, an Opus 5.5 agent
@@ -20,7 +20,7 @@ Native save reliability: synthetic cases on macOS reproduced loss through the pr
| 5 | **Content controls** landed (#381): `w:sdt` kept, shown and edited (Developer tab, Design Mode, Properties). Move tracking landed (#400): `w:moveFrom`/`w:moveTo` with their named ranges round-trip, a tracked cut and paste of a paragraph or sentence records a move, either end accepts/rejects both, green double strike/underline markup and "Moved from/to" balloons; still missing: drag-and-drop moves (no text drag yet) and move options in Track Changes Options. Formatting revisions landed (#41): `w:rPrChange`/`w:pPrChange`/`w:tblPrChange`/`w:trPrChange`/`w:tcPrChange`/`w:sectPrChange`/`w:numberingChange` round-trip, are recorded while tracking, accepted/rejected and shown in balloons; still missing: tracked style-definition changes and table/section changes made in WordCraft. Opaque custom XML backing stores now survive saves; binding evaluation remains absent. | B | Issues #41, #400; source grep | Legal and editorial review workflows; templates with form controls | 5–8 | [file-format-parity.md](file-format-parity.md) |
| 6 | **No native printing** on desktop: File › Print opens the PDF in the system viewer, which prints (#286); File › Print has a preview, page range, odd/even, copies and collate (#641); printer choice, duplex and pages per sheet are still the system's; web prints through the browser (#209) | B | `ui.print` → `apps/wordcraft/src/print.rs`; issue #15 | Everyone who prints | 5–8 | [hardware-parity.md](hardware-parity.md) |
| 7 | **Dialog depth**: 54 modal dialogs vs Word's ~100 | B | `Dialog` enum in `crates/ui-egui/src/dialogs.rs` | Power users reach for dialogs the ribbon alone doesn't cover | 11–19 | [ui-parity.md](ui-parity.md) |
| 8 | **Proofing is English only**; no language per run, no dictionaries for other languages | | `crates/proof` has one dictionary; Word ships 120 proofing tools; issues #25, #40, #100 | Every non-English writer | 25–45 | [localization-parity.md](localization-parity.md) |
| 8 | **Proofing covers English and Polish only**; no dictionaries, grammar or hyphenation for the other ~48 languages Word proofs (text in them is left unmarked) | | `crates/proof`: English and Polish, a language per run (2026-10-10); Word ships 120 proofing tools; issues #25, #40, #100 | Every writer outside English and Polish | 20–40 | [localization-parity.md](localization-parity.md) |
| 9 | **Password-protected files** can't be opened or saved | | #55; `docbin` detects and refuses RC4/XOR | Corporate and legal users | 6–10 | [file-format-parity.md](file-format-parity.md) |
| 10 | **Objects**: no group (#267 open), text in a freely rotated text box (free rotation and flips landed, #332; text direction, alignment and linked text boxes landed, #369; contour wrap and wrap side landed, #354), shape effects, WordArt 3-D effects and connector routing around shapes (WordArt effects and warps and glued connectors landed, #416), dragging members inside a Drawing Canvas (the canvas landed, #344) | | [parity-checklist.md](parity-checklist.md) (Shape Format 6/6, Layout › Group) | Newsletters, flyers, reports with diagrams | 20–30 | [target-app-parity.md](target-app-parity.md) |
| 11 | **Interface languages**: 7 of the 12 key languages missing (hi, ar, fr, id, ko, vi, pt-PT); no mirrored RTL interface; ~5% of strings still English in the translated catalogs | | [localization-parity.md](localization-parity.md) | Most of the world's writers | 50–80 | [localization-parity.md](localization-parity.md) |
@@ -118,6 +118,7 @@ The same gaps grouped the way the parity documents are, for agents working in on
| 2026-10-10 | trivial | #10: Drawing Canvas landed (#344); in-canvas dragging of members remains |
| 2026-10-10 | trivial | #29: long display equations wrap across lines (#326) |
| 2026-10-10 | trivial | #3: Windows lag on GPUs without a DirectX 12 driver fixed with an OpenGL fallback (#316) |
| 2026-10-10 | minor | #8: Polish proofing (spelling, grammar, hyphenation) and a proofing language per run landed; the gap is now the other languages (20–40 h) |
| 2026-10-10 | trivial | #6: desktop File › Print opens the system PDF viewer (#286) |
| 2026-10-10 | trivial | Table Properties dialog landed (#44); removed from the dialog-depth list |
| 2026-10-10 | trivial | Merged main: landed features removed from the feature-gap list |
docs/localization-parity.md
+32 -11
View File
@@ -1,12 +1,13 @@
# Localization parity with Microsoft Word
> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-11 · **Change:** minor (German interface catalog, #574; previously minor: Turkish interface added; previously major: first version; catalogs measured from `crates/ui-egui/src/i18n`, Word's languages from its bundle listing) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4
> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-11 · **Change:** minor (Polish interface; Polish proofing — spelling, grammar, hyphenation — and a proofing language per run; previously minor: Turkish interface added; previously major: first version; catalogs measured from `crates/ui-egui/src/i18n`, Word's languages from its bundle listing) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4
Interface languages, script support and proofing per language. How translation works (catalog
format, adding a language, clean-room rule): `crates/ui-egui/src/i18n/mod.rs`.
**Dimension: ~25% (estimated), 80–130 h to full for the twelve key languages** plus ~50 h for the
other ~18 languages Word ships, plus native-speaker review (human).
**Dimension: ~27% (estimated; 25% before Polish and per-run proofing languages), 75–120 h to full
for the twelve key languages** plus ~45 h for the other ~17 languages Word ships, plus
native-speaker review (human).
## What Word ships (measured by listing the bundle)
@@ -18,16 +19,20 @@ other ~18 languages Word ships, plus native-speaker review (human).
## What WordCraft ships (measured)
- **14 interface languages**: English plus `zh-hans`, `zh-hant`, `ja`, `uk`, `es`, `pt-br`,
Serbian `sr`/`sr-latn` (#250), Estonian `et`, Turkish `tr`, Russian `ru`, Italian `it` (#594) and German `de` (#574). Catalogs have 2,021–2,399 entries
each. A test (`every_tab_and_command_is_translated`) enforces that every ribbon tab, command
- **15 interface languages**: English plus `zh-hans`, `zh-hant`, `ja`, `uk`, `es`, `pt-br`,
Serbian `sr`/`sr-latn` (#250), Estonian `et`, Turkish `tr`, Russian `ru`, Italian `it` (#594), German `de` (#574) and Polish `pl`. Catalogs have 2,021–2,399 entries
each; Polish has every key of every other catalog (a test checks it). A test (`every_tab_and_command_is_translated`) enforces that every ribbon tab, command
label and ribbon location is translated in every language.
`pl`, `pl-PL` and `pl_PL.UTF-8` system locales pick Polish automatically.
- Of the 675 English strings a scan extracts from `tl!(…)` calls and command labels/locations, each
catalog covers **661 (98%)**. Some newer dialog and pane strings (Equation tab tooltips, Zotero,
Read Aloud, Backstage greetings) aren't wrapped in `tl!` yet, so they show in English in every
language: the true share of visible strings is **~93–95%** (estimated).
- Pending PRs: Norwegian Bokmål (#208) and Nynorsk (#210), Hebrew with RTL (#20).
- **Proofing: English only** (one dictionary, rule-based grammar, English hyphenation patterns).
- **Proofing: English and Polish**, chosen per run from its language (`w:lang`, resolved through
the styles; Review › Language and the status bar set it). Languages without proofing tools get
no spelling or grammar marks (as in Word without the language's tools) instead of being flagged
as English. Detail in [Proofing per language](#proofing-per-language).
- Document scripts: shaping via HarfRust (HarfBuzz port) for every script; bidi (UAX #9) and Arabic
shaping tested (#207); IME on all desktop platforms (winit fixes pending, #155–#164).
@@ -52,8 +57,8 @@ shared items below.
| Korean | ko | 0 | ❌ | Hangul ✅; macOS Korean IME fixes pending (#163, #164) | no | none | 3–5 |
| Vietnamese | vi | 0 | ❌ | Latin with stacked diacritics, untested | no | none | 3–4 |
Other shipped languages: **8** — Traditional Chinese (`zh-hant`), Ukrainian (`uk`), Serbian
Cyrillic and Latin (`sr`, `sr-latn`, #250), Estonian (`et`), Turkish (`tr`), Russian (`ru`) and Italian (`it`, #594), all partial.
Other shipped languages: **9** — Traditional Chinese (`zh-hant`), Ukrainian (`uk`), Serbian
Cyrillic and Latin (`sr`, `sr-latn`, #250), Estonian (`et`), Turkish (`tr`), Russian (`ru`), Italian (`it`, #594) and Polish (`pl`), all partial.
Estonian uses the existing fonts, follows `et-EE` system locales, and is available in Options or
through `ui.language` with `{"value":"et"}`. Its 1,307 entries cover the existing interface catalog.
Estonian spelling, grammar, dates and document templates are not added by the interface catalog.
@@ -62,12 +67,27 @@ entries. Command search folds the Turkish dotted and dotless I (`italik` finds
finds Yazı Tipi) while English labels and ASCII command ids keep their usual matching. Turkish
proofing is not included; terminology and checks: [`localization-tr.md`](localization-tr.md).
Polish is the only one besides English with proofing tools, and a Polish interface writes new
documents in Polish (`pl-PL`).
## Proofing per language
| Language | Spelling | Suggestions | Grammar | Hyphenation |
|---|---|---|---|---|
| English (`en`, any region; text with no language) | Moby word list (public domain, ~160k words) with inflection rules | edit distance | repeated word, `a`/`an`, sentence capital, space before punctuation, double space | Moby dictionary, our Liang patterns, heuristic |
| Polish (`pl`) | The Hunspell `pl_PL` dictionary installed on the system (OS package, Homebrew, LibreOffice, or the `WORDCRAFT_DICTIONARIES` folder; not bundled, so no Polish spelling marks without it), read at runtime by a Hunspell-style affix engine (`crates/proof/src/affix.rs`); names capitalized, compounds part by part, `Kennedy'ego` | missing diacritics first (`zolw` → `żółw`), the dictionary's confusion table (`żeka` → `rzeka`, `chuśtawka` → `huśtawka`), missing space (`napewno` → `na pewno`), one typo, then a stem scan within two typos | Polish messages: repeated word, space before punctuation, double space, sentence capital (knows `np.`, `r.`, initials, ordinals, dialogue dashes), missing comma before `że`/`który`/`ponieważ`/`gdyż`/`aby`/`żeby`/`jeśli`/`jeżeli`/`gdyby`/`lecz` (not after `mimo`, `chyba`, `tak`, prepositions…), `we`/`ze` (`we wtorek`, `ze szkoły`), doubled punctuation | TeX Polish patterns (Kołodziejska, Jackowski, Ryćko; MIT option) with their 20 exception words; a Polish syllable heuristic for letters outside the alphabet |
| Any other | — (no marks) | — | — | the English patterns, as before |
Measured with the SJP.PL dictionary: every one of the ~4.5 million forms an independent expansion of its affix file
produces is accepted, and 200,000 one-edit mutations that aren't words are all rejected; a check
takes ~1 µs, a suggestion list 1–35 ms (release build). Not yet: other languages' dictionaries,
a thesaurus for Polish, Word's style and clarity refinements, Polish-specific AutoCorrect.
Italian (`it`, #594) follows `it`, `it-IT` and `it-CH` system locales. Its catalog has every key of
the others plus 51 strings that are wrapped in `tl!` but were in no catalog yet (File › Info
properties, the File › Home greeting, list levels, Equation tab gallery and tooltips). Ribbon tabs,
groups and command names use the names Italian Word users know (feature names, as `AGENTS.md`
allows); all other text is written from the meaning of the English. No Italian proofing yet.
## Shared work (not in the per-language hours)
| Item | Hours | Notes |
@@ -75,13 +95,14 @@ allows); all other text is written from the meaning of the English. No Italian p
| Wrap the remaining English literals in `tl!` and translate them in every catalog | 4–6 | Makes the six partial languages full except script items |
| Mirrored (right-to-left) interface for Arabic and Hebrew | 15–25 | Ribbon, panes, dialogs, rulers |
| Vertical text, ruby, document grid (Chinese, Japanese, Korean) | counted in [layout-parity.md](layout-parity.md) | |
| Proofing dictionaries, grammar and hyphenation for the 11 non-English key languages | 25–45 | Open dictionaries only (licence check per dictionary); issues #25, #40, #100 |
| Proofing dictionaries, grammar and hyphenation for the 11 non-English key languages | 20–40 | Open dictionaries only (licence check per dictionary); issues #25, #40, #100. Per-run languages, the Hunspell-style affix engine and the TeX pattern reader landed with Polish, so a language with an open Hunspell dictionary and hyphenation patterns is mostly data plus grammar rules |
| Native-speaker review of every catalog | human | None reviewed yet |
## Revision history
| Date | Change | Summary |
|---|---|---|
| 2026-10-11 | minor | Polish interface (13 languages) and Polish proofing: SJP.PL spelling with suggestions, Polish grammar rules, TeX Polish hyphenation; proofing language per run; dimension ~25% → ~27% |
| 2026-10-11 | minor | German interface catalog (#574): 2,021 entries covering the full zh-hans key set; locale, saved-preference and bundled-font tests; no German proofing |
| 2026-10-11 | minor | Italian interface catalog (#594), locale selection, saved preference and font coverage checks; count labels put the noun first so they read right at 1 |
| 2026-10-11 | minor | Font dialog sets the Asian text font (`w:eastAsia`) and the Latin text font separately; the Asian list holds the installed Chinese, Japanese and Korean fonts by their interface-language names (#505) |
docs/roadmap.md
+3 -2
View File
@@ -1,6 +1,6 @@
# WordCraft roadmap: milestones and what's next
> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-11 · **Change:** trivial (Draw: Lasso Select, Add Pen, Ink to Shape and Ink Replay landed, #395; previously trivial: M7: formatting revisions landed, #41; previously trivial: M5: Draw Table and Eraser landed, #303; previously minor: beta distance restated: ~15 points, ~150–230 h; previously: alpha gate checked: six core workflows pass; previously major: milestones moved here from ROADMAP.md; Current focus set from the re-measured gaps) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4
> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-11 · **Change:** minor (Polish interface and Polish proofing; previously trivial: M7: formatting revisions landed, #41; previously trivial: M5: Draw Table and Eraser landed, #303; previously minor: beta distance restated: ~15 points, ~150–230 h; previously: alpha gate checked: six core workflows pass; previously major: milestones moved here from ROADMAP.md; Current focus set from the re-measured gaps) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4
Forward-looking plan. The summary is [`ROADMAP.md`](../ROADMAP.md); the ranked work list is
[`gaps.md`](gaps.md); the numbers are in [`target-app-parity.md`](target-app-parity.md). Hours are
@@ -60,7 +60,7 @@ partial rows are fidelity gaps (beta work), not broken workflows. WordCraft stay
| M12 | Formats breadth (PDF, ODT, RTF, HTML, MD, TXT, LaTeX, `.doc` import) | done (first versions); `.doc` write, encryption, minor formats missing | 60–100 h |
| M13 | Performance budgets | on track (1.4 ms relayout); large real-world documents unmeasured | 10–20 h |
| M14 | 1.0 polish, packaging, signing | releases v0.1.0–v0.4.0 published (signed macOS, Windows MSI x64/x86/arm64, Linux, FreeBSD, web) | — |
| M15 | Localization (12 key languages, RTL interface, proofing languages) | 11 UI languages, English proofing | 80–130 h |
| M15 | Localization (12 key languages, RTL interface, proofing languages) | 15 UI languages; English and Polish proofing, a proofing language per run | 75–120 h |
| M16 | Right-to-left and East Asian typography | RTL paragraphs done (#207) | 30–45 h |
## Next after beta
@@ -72,6 +72,7 @@ languages, screen-reader access, ecosystem decisions (VBA, add-ins, cloud), an i
| Date | Change | Summary |
|---|---|---|
| 2026-10-11 | trivial | M15: Polish interface and Polish proofing landed |
| 2026-10-11 | trivial | M3: icons landed (#309) |
| 2026-10-11 | trivial | M11: contour (Tight/Through) wrap done (#354) |
| 2026-10-11 | trivial | M6: the eight missing bibliography styles landed (#383) |
docs/target-app-parity.md
+4 -3
View File
@@ -1,6 +1,6 @@
# WordCraft parity with Microsoft Word
> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-11 · **Change:** trivial (table borders and the resize handle drag on the page, #217; previously trivial: all 12 bibliography styles, #383; previously trivial: formatting revisions landed, #41; previously trivial: Draw Table and Eraser landed, #303; previously minor: readiness table with hours per audience; full number back to the additive weighted sum, 60%; mainstream and essentials numbers added; alpha gate checked; previously major: full re-measure against Word for Mac 16.113.4; replaces the parity tables that lived in ROADMAP.md) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4
> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-11 · **Change:** minor (Polish interface and Polish proofing; previously trivial: English spelling knows common modern words, #520; previously trivial: all 12 bibliography styles, #383; previously trivial: formatting revisions landed, #41; previously trivial: Draw Table and Eraser landed, #303; previously minor: readiness table with hours per audience; full number back to the additive weighted sum, 60%; mainstream and essentials numbers added; alpha gate checked; previously major: full re-measure against Word for Mac 16.113.4; replaces the parity tables that lived in ROADMAP.md) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4
The authoritative assessment of how close WordCraft is to Microsoft Word. [`ROADMAP.md`](../ROADMAP.md)
summarizes it; [`gaps.md`](gaps.md) itemizes every shortfall; the deep checklists are
@@ -66,7 +66,7 @@ full parity for the area.
| Footnotes and endnotes | 3 | 60% | 60% | 3–5 | Placed and editable, numbers from Word files (#103); long notes continue onto the next page under the continuation separator (#352). Footnote and Endnote dialog with options per section: below text, end of section, number format, start at, restart each section or page, custom mark, convert (#385). |
| References (TOC, citations, bibliography, captions, index, TOA) | 5 | 65% | 60% | 12–20 | Working first versions, Zotero integration (#189), Zotero/Mendeley `ADDIN` fields round-trip. Caption, Index, Mark Entry, Custom TOC and Create/Edit Source dialogs; updates honour the TOC (`\o \u \t \f \n \p \h`), INDEX (`\c \e \h \r`), XE (`\b \i \t \r`) and SEQ/STYLEREF chapter switches (#402). All 12 of Word's bibliography styles (APA, MLA, Chicago, IEEE, Harvard, ISO 690 author-date and numerical, Turabian, GB/T 7714, GOST name and title sort, SIST02; #383); bibliography sources not yet in DOCX (#169 open). |
| Review (proofing UI, comments, track changes, compare, protect) | 8 | 72% | 75% | 15–25 | Comment balloons, tracked insert/delete incl. paragraph marks (#244, #125), Compare / Combine with comparison settings (case, white space, formatting, tables, word or character level, #412), Restrict Editing pane with formatting and editing restrictions, exceptions for everyone and a password (#412). Formatting revisions (#41): kept in DOCX, recorded while tracking, accepted/rejected, "Formatted: …" balloons and change bars. Moves (#400): kept in DOCX, recorded on a tracked cut and paste, accepted/rejected at either end, green markup and "Moved from/to" balloons. Show revisions in balloons takes tracked deletions in the body out of the text into "Deleted: …" balloons (insertions stay inline, as in Word; deletions in headers, footers, notes and text boxes, and in print and PDF, stay inline). Missing: drag-and-drop moves, insert/delete balloons, Translate, Block Authors, modern comment threads/mentions. Lowered: these gaps are confirmed by source (no `rPrChange` anywhere). |
| Proofing (spelling, grammar, thesaurus, languages) | 5 | 35% | (in Review) | 25–45 | English only: one dictionary (Moby list plus our supplement of modern words such as "desktop", #520), a rule-based grammar checker. Word ships 120 proofing tools for ~50 languages plus Editor (style refinements, similarity). Issues #25, #40, #100. |
| Proofing (spelling, grammar, thesaurus, languages) | 5 | 40% | (in Review) | 20–40 | English and Polish, chosen per run from its language (`w:lang`): English word list (Moby plus our supplement of modern words such as "desktop", #520) and rule-based grammar; Polish against the installed Hunspell dictionary through a Hunspell-style affix engine, Polish grammar rules and TeX hyphenation patterns (2026-10-10). Other languages are left unmarked. Word ships 120 proofing tools for ~50 languages plus Editor (style refinements, similarity). Issues #25, #40, #100. |
| Mailings | 2 | 70% | 70% | 6–10 | Mail merge from CSV, fields, rules (IF, SKIPIF, NEXT, NEXTIF, evaluated alike in preview and merge), fields in headers/footers (one section per letter), preview, envelopes, labels. Select Recipients takes a typed list or CSV (#247); no Excel/Outlook/Contacts data sources, no email merge. |
| Pictures, shapes, text boxes, WordArt | 6 | 60% | 60% | 20–30 | Insert, crop, recolour, styles, floating placement like Word (#136), wrap (square/top-bottom/behind/front), text boxes edit and overflow (#46), VML text boxes (#242), free rotation and flips with a rotation handle (#332), WordArt: text fill (solid, gradient, none), outline, shadow, glow and reflection with our own styles, and Transform warps along arch, circle, wave, slant, inflate/deflate, chevron and triangle (other presets kept and drawn flat), connectors (straight, elbow, curved, arrowheads) glued to shapes' connection sites and re-routed when those move (#416). Missing: tight/through contour wrap, rotated text-box text, group (#267 open), shape effects, WordArt 3-D/bevel and editing text in its bent form, connector routing around shapes, Drawing Canvas, Icons, online pictures, screenshot. |
| Mailings | 2 | 70% | 70% | 6–10 | Mail merge from CSV, fields, rules (IF, SKIPIF, NEXT, NEXTIF, evaluated alike in preview and merge), fields in headers/footers (one section per letter), preview, envelopes, labels. Select Recipients takes a typed list, CSV (#247) or a sheet of an Excel (.xlsx/.xlsm) or OpenDocument (.ods) workbook, with Select Table to pick the sheet (#334); no Outlook/Contacts/Access data sources, no email merge. |
@@ -195,7 +195,7 @@ Word. Their weighted sum is the **full ready for real work** number.
| UI/UX fidelity | 10% | 65% | 65–100 | [ui-parity.md](ui-parity.md) | Ribbon, keytips, mini-toolbar, Backstage, 125 shortcuts; 16 modal dialogs against Word's ~100; no ribbon/keyboard customization |
| Stability | 10% | 50% | 25–45 | [gaps.md](gaps.md) | Never-crash standard and hostile-param fuzzing in place, but open reports of a startup crash on Intel UHD (#170), freezes (#77, #59, #31) and Windows write failure without admin (#218); no soak test; 5 days of field history |
| Performance | 5% | 75% | 10–20 | [layout-parity.md](layout-parity.md) | 61 ms cold layout and 1.4 ms relayout on a 188-page document (measured 2026-10-06, not re-run); font picker freezes fixed in #233; no large real-world documents measured |
| Localization | 4% | 25% | 80–130 + native review | [localization-parity.md](localization-parity.md) | 9 UI languages (Word 30); 4 of the 12 key languages partly done; English-only proofing |
| Localization | 4% | 27% | 75–120 + native review | [localization-parity.md](localization-parity.md) | 13 UI languages (Word 30); 4 of the 12 key languages partly done; English and Polish proofing |
| Platforms | 3% | 95% | 5–10 | [ROADMAP.md](../ROADMAP.md) | macOS (universal, signed), Windows x64/x86/arm64, Linux (AppImage/deb/rpm/Flatpak), FreeBSD, web. Beyond Word on Linux, BSD and the web; Word's iPad/iPhone/Android apps are out of scope |
| Hardware | 1% | 40% | 20–40 | [hardware-parity.md](hardware-parity.md) | No native printing on desktop, no pen/ink, no dictation; GPU drawing and HiDPI fine. Ink hours are in the Features row (Draw tab) |
| Ecosystem and plugins | 2% | 15% | 40–80 | [gaps.md](gaps.md) | Zotero built in, macros record/play commands, `.docm` macros preserved (#172). No VBA execution, no Office add-ins, no EndNote/Mendeley plugins, no cloud storage or co-authoring |
@@ -251,6 +251,7 @@ can be used without committing Word output.
| Date | Change | Summary |
|---|---|---|
| 2026-10-11 | minor | Polish interface (13 UI languages) and Polish proofing with per-run languages: Proofing 35% → 40% (25–45 h → 20–40 h), Localization 25% → 27%; feature depth and readiness unchanged at this precision |
| 2026-10-11 | trivial | Tables: column and row borders and the resize handle drag on the page (#217); Tables row and interaction-fidelity evidence updated, no percentage change |
| 2026-10-11 | trivial | Review: Show revisions in balloons takes tracked deletions in the body out of the text into "Deleted: …" balloons (it drew like the comments-and-formatting mode before); Review row updated, no percentage change |
| 2026-10-11 | trivial | Insert › SmartArt (#497): charts/SmartArt row updated; catalog 429/451 → 437/458 (95.4%); no percentage change |
docs/ui-parity.md
+3 -2
View File
@@ -1,6 +1,6 @@
# UI parity with Microsoft Word
> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-11 · **Change:** trivial (table borders and the resize handle drag on the page, #217; previously major: first version; ribbon, dialogs, shortcuts and on-canvas interaction measured from `crates/ui-egui` and the command registry) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4
> **Last reviewed:** 2026-10-10 · **Last updated:** 2026-10-11 · **Change:** minor (status bar language follows the caret and sets the proofing language; Review › Language is a menu; previously trivial: Customize Ribbon and the Quick Access Toolbar, #379; previously: major, first version; ribbon, dialogs, shortcuts and on-canvas interaction measured from `crates/ui-egui` and the command registry) · **Target:** Microsoft Word (Microsoft 365) for Mac 16.113.4
How WordCraft looks and feels next to Word: ribbon, dialogs, panes, keyboard, mouse and touch.
Typing behaviour has its own checklist ([`typing-parity.md`](typing-parity.md)). All assets are our
@@ -29,8 +29,8 @@ own (`AGENTS.md`: no Microsoft iconography, ever).
| Selection: word/sentence/paragraph clicks, dragging after a double-click/triple-click/Ctrl+click extends by whole words/paragraphs/sentences (#549), Shift extend, F8 extend mode, column (Alt+drag, #237) | ✅ | ✅ | 90% | 1 |
| Zoom: Ctrl+scroll, pinch (#179), zoom buttons (#232), pages side by side when zoomed out (#251) | ✅ | ✅ | 85% | — |
| Touchpad smooth/inertial scroll | ✅ (#252) | ✅ | 90% | — |
| Status bar (page x of y, words, language, view buttons, zoom slider) | ✅ (the language is the caret's; a click sets the proofing language) | ✅ | 85% | — |
| Caret motion: keys and typing glide the caret to its new place (100 ms, `cubic-bezier(0, 0, 0, 1)`), the blink fades (530 ms on and off, 100 ms fades); clicks and scripted moves don't glide. File › Options › Advanced › Animation (`ui.caretAnimation`) turns it off | ✅ as measured from Word for Windows (`caret_glide.rs`); zoom levels other than 100%, modifier keys and off-screen moves not measured | ✅ on Windows; Word for Mac not measured | 90% | 1–2 |
| Status bar (page x of y, words, language, view buttons, zoom slider) | ✅ | ✅ | 80% | — |
| Interface themes (light, dark, system #249, follows OS changes live #311), dark page separate from the interface theme (#194, #312) | ✅ | ✅ | 90% | — |
| Interface size (File › Options › General, 80–200 %, on top of the system display scale; `ui.interfaceSize`) (#475) | ✅ | system display scaling only | 100% | — |
| Animation effects (File › Options › Accessibility: Use system setting, On, Off; `ui.animations`). Off: a wheel notch scrolls at once; Ctrl+wheel zooms in whole steps, each drawn finished in its own frame (a notch at once; touchpad pinches and high-resolution wheels in steps of about 10% rather than gliding); menus, buttons and the scroll to the caret neither fade nor glide | ✅ "Use system setting" reads Windows (Animation effects), macOS (Reduce motion), the Linux/BSD desktops (settings portal: the cross-desktop reduced-motion preference, GNOME's switch, KDE's animation speed; the GTK and KDE settings files where no portal answers) and browsers (`prefers-reduced-motion`), and follows a change while running. Run on Windows and against a stand-in portal on Linux; not yet tried on a Mac or a real GNOME or KDE session | not compared | 90% | 0.5–1 |
@@ -87,6 +87,7 @@ own (`AGENTS.md`: no Microsoft iconography, ever).
| 2026-10-10 | trivial | System interface theme keeps following OS appearance changes (no longer pins the macOS window); Dark page no longer darkens the interface (#311, #312) |
| 2026-10-11 | trivial | Insert › Icons picker: search, category chips, icons drawn live from the library's path data; Picture Format › Icon Color (#309) |
| 2026-10-10 | trivial | AutoSave switch greyed out with a tooltip saying why where AutoSave can't save; Save As from it (#196, #176) |
| 2026-10-10 | minor | Status bar language: the caret's language, a click opens the proofing-language menu (also Review › Language); context menu items translated |
| 2026-10-10 | trivial | Table Properties dialog and Table Layout Height/Width boxes (#44) |
| 2026-10-10 | minor | Merged main: Style Inspector, Column Selection, zoom buttons, inertial scrolling, File Info landed |
| 2026-10-10 | major | First version: ribbon, dialog, shortcut and interaction inventory |