harald/ink-notes
2
1 Fork 0
Code Issues 3 Pull Requests Actions Packages Projects Releases 64 Wiki Activity

[Feature] Inline Handwritten Math Recognition & Auto-Evaluation ("2^4 = 16") #46

New Issue
Open
opened 2026-10-25 14:01:11 +01:00 by harald · 2 comments
harald commented 2026-10-25 14:01:11 +01:00
Owner
Copy link
Copy source

Summary

When taking math or scientific lecture notes with a digital stylus, it is cumbersome to leave the app to calculate numbers or expand formulas.

We should add an inline handwritten math helper: when a user writes an equation ending with an = sign (e.g., 2^4 = or (x + 3)^2 =), the app should recognize the handwritten expression, calculate or simplify the result, and display it next to the = sign.

User Experience (UX) Flow

  1. The user writes a formula with the stylus: e.g. 2^4 =.
  2. Upon completing the = sign (and a ~500ms pen pause), the app detects the trigger.
  3. A subtle preview chip or ghost text appears to the right of =: 16.
  4. Tapping the preview inserts the result permanently into the document.

Proposed Architecture (Approach B: Canvas Crop + Pure-Rust Inference)

To guarantee full OS independence and native support for Linux (Zorin OS) without external C/C++ runtime dependencies (.so / .dll), the feature should use:

  1. Stroke Bounding Box & Rasterizer:

    • Detect the strokes preceding the trailing = sign.
    • Render the selected vector strokes into an in-memory grayscale image buffer (fixed height of 64px, dynamic width scaled to maintain aspect ratio).

  2. Handwriting Recognition (tract-onnx):

    • Run a tiny, pre-trained math OCR model (CRNN / mini-ViT trained on math datasets like CROHME) embedded as an ONNX file.
    • Execute inference strictly on CPU using tract-onnx (100% pure Rust, zero system dependencies, fast SIMD execution).
    • Decode the output tensor into a clean math string (e.g., "2^4").

  3. Deterministic Math Evaluator:

    • Arithmetic: Evaluate numeric expressions using a pure-Rust parser like evalexpr (handles powers ^, basic operations, trig).
    • Symbolic Fallback (Binomials/Formulas): If variables like x or a are present, use a symbolic engine (e.g. kalk or symbolic_polynomials) to expand expressions like (x + 3)^2 \rightarrow x^2 + 6x + 9.

Technical Constraints

  • Zero C/C++ dynamic linking: Must compile using standard cargo build on Linux (Zorin OS / Ubuntu), Windows, and macOS.
  • Offline & Private: All inference and evaluation must run 100% locally on CPU without external API calls.
  • Low Latency: Recognition and evaluation combined must execute in < 25ms on modern laptop CPUs so note-taking remains fluid.

Acceptance Criteria / Tasks

  • Implement trailing = stroke gesture detection.
  • Create an in-memory stroke rasterizer (64px height grayscale tensor).
  • Integrate tract-onnx to load an embedded math OCR model and run CPU inference.
  • Add the math evaluation pipeline (evalexpr for arithmetic, symbolic fallback for variables).
  • Render the computed result inline next to the = sign with an interaction to confirm/bake into ink.
  • Add unit tests for expression decoding and mathematical evaluation.
### Summary When taking math or scientific lecture notes with a digital stylus, it is cumbersome to leave the app to calculate numbers or expand formulas. We should add an **inline handwritten math helper**: when a user writes an equation ending with an `=` sign (e.g., `2^4 =` or `(x + 3)^2 =`), the app should recognize the handwritten expression, calculate or simplify the result, and display it next to the `=` sign. --- ### User Experience (UX) Flow 1. The user writes a formula with the stylus: e.g. `2^4 =`. 2. Upon completing the `=` sign (and a ~500ms pen pause), the app detects the trigger. 3. A subtle preview chip or ghost text appears to the right of `=`: `16`. 4. Tapping the preview inserts the result permanently into the document. --- ### Proposed Architecture (Approach B: Canvas Crop + Pure-Rust Inference) To guarantee full **OS independence** and native support for Linux (Zorin OS) without external C/C++ runtime dependencies (`.so` / `.dll`), the feature should use: 1. **Stroke Bounding Box & Rasterizer:** - Detect the strokes preceding the trailing `=` sign. - Render the selected vector strokes into an in-memory grayscale image buffer (fixed height of `64px`, dynamic width scaled to maintain aspect ratio). 2. **Handwriting Recognition (`tract-onnx`):** - Run a tiny, pre-trained math OCR model (CRNN / mini-ViT trained on math datasets like CROHME) embedded as an ONNX file. - Execute inference strictly on CPU using [`tract-onnx`](https://crates.io/crates/tract-onnx) (100% pure Rust, zero system dependencies, fast SIMD execution). - Decode the output tensor into a clean math string (e.g., `"2^4"`). 3. **Deterministic Math Evaluator:** - **Arithmetic:** Evaluate numeric expressions using a pure-Rust parser like [`evalexpr`](https://crates.io/crates/evalexpr) (handles powers `^`, basic operations, trig). - **Symbolic Fallback (Binomials/Formulas):** If variables like `x` or `a` are present, use a symbolic engine (e.g. [`kalk`](https://crates.io/crates/kalk) or [`symbolic_polynomials`](https://crates.io/crates/symbolic_polynomials)) to expand expressions like `(x + 3)^2` $\rightarrow$ `x^2 + 6x + 9`. --- ### Technical Constraints - **Zero C/C++ dynamic linking:** Must compile using standard `cargo build` on Linux (Zorin OS / Ubuntu), Windows, and macOS. - **Offline & Private:** All inference and evaluation must run 100% locally on CPU without external API calls. - **Low Latency:** Recognition and evaluation combined must execute in < 25ms on modern laptop CPUs so note-taking remains fluid. --- ### Acceptance Criteria / Tasks - [ ] Implement trailing `=` stroke gesture detection. - [ ] Create an in-memory stroke rasterizer (`64px` height grayscale tensor). - [ ] Integrate `tract-onnx` to load an embedded math OCR model and run CPU inference. - [ ] Add the math evaluation pipeline (`evalexpr` for arithmetic, symbolic fallback for variables). - [ ] Render the computed result inline next to the `=` sign with an interaction to confirm/bake into ink. - [ ] Add unit tests for expression decoding and mathematical evaluation.
harald commented 2026-10-25 14:04:20 +01:00
Author
Owner
Copy link
Copy source

Review against the code (planned for the future, not started)

Platform

  • ink-notes runs on Linux only (Wayland/X11, octotablet for the pen), with Android planned in #42–#45. Windows and macOS are out of scope, so "zero C/C++ dynamic linking" only has to hold for Linux and Android. That is still a good rule; tract and the evaluator are pure Rust.

Recognition: use the strokes, not a raster crop

  • Every stroke is stored as vector samples with position, pressure and a timestamp (StylusSample). That makes online recognition possible: the order of the strokes and the pen direction are known. This is more accurate and much cheaper than recognising a 64 px image.
  • Rasterising would also be extra work. The current CPU rasteriser goes through PDF and hayro (page_export.rs), which is far too slow for < 25 ms. A dedicated stroke-to-tensor rasteriser would be needed.
  • A suggested pipeline that fits the 25 ms budget:
    1. Group strokes into symbols, by overlap and gap, plus time.
    2. Classify each symbol with a small CNN or MLP on the resampled stroke points: digits, + − · × : / ( ) = ^ x y a b. The model is a few hundred KB, runs on tract-onnx or as hand-written matrix multiplications, and needs well under 1 ms per symbol.
    3. Recognise the structure from the bounding boxes: baseline, superscript (2⁴ is written as a raised 4, there is no ^ on paper), fraction bars.
  • A full image-to-LaTeX model trained on CROHME (CRNN/ViT encoder-decoder such as BTTR or CoMER) would be tens of MB, and decoding it autoregressively is unlikely to reach < 25 ms on a laptop CPU. A CRNN with CTC reads only one line and cannot represent superscripts. Also to check: whether a suitable pretrained model and the CROHME data have licenses that allow shipping the model in the app.

Evaluation

  • evalexpr uses ^ for powers and has math::sin etc. That covers arithmetic. But the recognised string has to be normalised first:
    • the German decimal comma (2,5);
    • ·, × and : for multiply and divide;
    • implicit multiplication (2x, 3(x+1)).
  • For the symbolic part ((x + 3)^2): as far as I know, kalk is a calculator with variables but does not expand polynomials, so that needs checking. A small polynomial module of our own (sums of monomials with rational coefficients; + − · and integer powers) is about 200 lines and fully testable. I would prefer it to an extra dependency.

Triggering and conflicts

  • Draw-and-hold already snaps a stroke to a shape when the pen is held still for over 400 ms. The two bars of = may therefore arrive as snapped lines and not as ink, so detection must accept both.
  • = = two short, roughly horizontal, parallel strokes of similar length, drawn one after the other. The trigger fires after the pen is lifted and 500 ms pass without a new stroke. A new stroke cancels it.
  • Expression strokes = strokes left of the = on the same line (vertical overlap with a band around the =), with no large gap, written in this session.
  • The feature should be off by default (toggle in ☰ settings), so that an = in normal notes never gets a suggestion it did not ask for.

Showing and inserting the result

  • The preview chip can be an egui overlay at the page position. It must not catch palm or finger touches; pen tap and click only.
  • Insert the result as a text object, using the existing text tool (insert_text_on_page, in pen colour and matching the height of the handwriting), as one undo step. It stays editable and is exported to PDF like other text. Turning it into ink would need a handwriting font, so it could come later as an option.

Tests

  • Symbol grouping and structure from synthetic strokes, normalising the string, and the evaluator and polynomial expansion.
  • A small set of real handwriting samples (Pia's?) as a regression test for the classifier.
**Review against the code** (planned for the future, not started) **Platform** - ink-notes runs on Linux only (Wayland/X11, octotablet for the pen), with Android planned in #42–#45. Windows and macOS are out of scope, so "zero C/C++ dynamic linking" only has to hold for Linux and Android. That is still a good rule; tract and the evaluator are pure Rust. **Recognition: use the strokes, not a raster crop** - Every stroke is stored as vector samples with position, pressure and a timestamp (`StylusSample`). That makes *online* recognition possible: the order of the strokes and the pen direction are known. This is more accurate and much cheaper than recognising a 64 px image. - Rasterising would also be extra work. The current CPU rasteriser goes through PDF and hayro (`page_export.rs`), which is far too slow for < 25 ms. A dedicated stroke-to-tensor rasteriser would be needed. - A suggested pipeline that fits the 25 ms budget: 1. **Group strokes into symbols**, by overlap and gap, plus time. 2. **Classify each symbol** with a small CNN or MLP on the resampled stroke points: digits, `+ − · × : / ( ) = ^ x y a b`. The model is a few hundred KB, runs on tract-onnx or as hand-written matrix multiplications, and needs well under 1 ms per symbol. 3. **Recognise the structure** from the bounding boxes: baseline, superscript (`2⁴` is written as a raised `4`, there is no `^` on paper), fraction bars. - A full image-to-LaTeX model trained on CROHME (CRNN/ViT encoder-decoder such as BTTR or CoMER) would be tens of MB, and decoding it autoregressively is unlikely to reach < 25 ms on a laptop CPU. A CRNN with CTC reads only one line and cannot represent superscripts. Also to check: whether a suitable pretrained model and the CROHME data have licenses that allow shipping the model in the app. **Evaluation** - `evalexpr` uses `^` for powers and has `math::sin` etc. That covers arithmetic. But the recognised string has to be normalised first: - the German decimal comma (`2,5`); - `·`, `×` and `:` for multiply and divide; - implicit multiplication (`2x`, `3(x+1)`). - For the symbolic part (`(x + 3)^2`): as far as I know, `kalk` is a calculator with variables but does not expand polynomials, so that needs checking. A small polynomial module of our own (sums of monomials with rational coefficients; `+ − ·` and integer powers) is about 200 lines and fully testable. I would prefer it to an extra dependency. **Triggering and conflicts** - Draw-and-hold already snaps a stroke to a shape when the pen is held still for over 400 ms. The two bars of `=` may therefore arrive as snapped lines and not as ink, so detection must accept both. - `=` = two short, roughly horizontal, parallel strokes of similar length, drawn one after the other. The trigger fires after the pen is lifted and 500 ms pass without a new stroke. A new stroke cancels it. - Expression strokes = strokes left of the `=` on the same line (vertical overlap with a band around the `=`), with no large gap, written in this session. - The feature should be **off by default** (toggle in ☰ settings), so that an `=` in normal notes never gets a suggestion it did not ask for. **Showing and inserting the result** - The preview chip can be an egui overlay at the page position. It must not catch palm or finger touches; pen tap and click only. - Insert the result as a **text object**, using the existing text tool (`insert_text_on_page`, in pen colour and matching the height of the handwriting), as one undo step. It stays editable and is exported to PDF like other text. Turning it into ink would need a handwriting font, so it could come later as an option. **Tests** - Symbol grouping and structure from synthetic strokes, normalising the string, and the evaluator and polynomial expansion. - A small set of real handwriting samples (Pia's?) as a regression test for the classifier.
harald commented 2026-10-28 15:09:49 +01:00
Author
Owner
Copy link
Copy source

Erste Version in v0.41.0, experimentell und standardmäßig aus.

  • Einschalten: ☰ → Eingabe → „Formeln nach „=“ ausrechnen“.
  • Ablauf: Nach einem handgeschriebenen = und kurzer Pause erscheint daneben das Gelesene samt Ergebnis, z. B. 2⁴ = 16 oder (x + 3)² = x² + 6x + 9.
    • Antippen mit dem Stift oder Klicken fügt das Ergebnis als Text ein, in Stiftfarbe und Handschrifthöhe.
    • Finger und Handballen lösen nichts aus.
  • Gelesen werden: Ziffern, + − · : /, Klammern, Hochzahlen, Bruchstriche, Dezimalkomma sowie x, y, a, b.
  • Rechnen: Zahlen werden exakt gerechnet, Ausdrücke mit Variablen ausmultipliziert.
  • Erkennung: Punktwolken-Vorlagen ($P), offline, ohne trainiertes Modell, etwa 4 ms pro Formel.

Bisher nur mit künstlicher Handschrift getestet. Bitte mit echter Handschrift ausprobieren und melden, welche Zeichen falsch gelesen werden. Daraus entsteht der nächste Schritt: eigene Vorlagen für die eigene Handschrift.

Erste Version in **v0.41.0**, experimentell und standardmäßig aus. - **Einschalten:** ☰ → Eingabe → „Formeln nach „=“ ausrechnen“. - **Ablauf:** Nach einem handgeschriebenen `=` und kurzer Pause erscheint daneben das Gelesene samt Ergebnis, z. B. `2⁴ = 16` oder `(x + 3)² = x² + 6x + 9`. - Antippen mit dem Stift oder Klicken fügt das Ergebnis als Text ein, in Stiftfarbe und Handschrifthöhe. - Finger und Handballen lösen nichts aus. - **Gelesen werden:** Ziffern, `+ − · : /`, Klammern, Hochzahlen, Bruchstriche, Dezimalkomma sowie `x`, `y`, `a`, `b`. - **Rechnen:** Zahlen werden exakt gerechnet, Ausdrücke mit Variablen ausmultipliziert. - **Erkennung:** Punktwolken-Vorlagen ($P), offline, ohne trainiertes Modell, etwa 4 ms pro Formel. Bisher nur mit künstlicher Handschrift getestet. Bitte mit echter Handschrift ausprobieren und melden, welche Zeichen falsch gelesen werden. Daraus entsteht der nächste Schritt: eigene Vorlagen für die eigene Handschrift.
Sign in to join this conversation.
No Branch/Tag Specified
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
pia harald (Harald Pansi)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: harald/ink-notes#46