feat(storage): Formatversionierung, Migrations-Backups und Fixture-Tests
This commit is contained in:
+38
-20
@@ -1,6 +1,6 @@
|
||||
# Dateiformat & Speicherstruktur von ink-notes
|
||||
|
||||
Dieses Dokument beschreibt die Speicherstruktur und die JSON-Dateiformate, mit denen `ink-notes` Dokumente auf dem Dateisystem ablegt.
|
||||
Dieses Dokument beschreibt die Speicherstruktur, die JSON-Dateiformate und die Versionshistorie, mit denen `ink-notes` Dokumente auf dem Dateisystem ablegt.
|
||||
|
||||
---
|
||||
|
||||
@@ -13,7 +13,9 @@ Notizbücher werden standardmäßig unter dem XDG-Datenpfad gespeichert:
|
||||
└── notebooks/
|
||||
├── <notebook_id_1>/
|
||||
│ ├── manifest.json
|
||||
│ ├── manifest.json.v0.bak (optional, Migrations-Backup)
|
||||
│ ├── page_1.json
|
||||
│ ├── page_1.json.corrupt-1775460000 (optional, Quarantäne bei Parsefehlern)
|
||||
│ ├── page_2.json
|
||||
│ └── ...
|
||||
└── <notebook_id_2>/
|
||||
@@ -29,19 +31,20 @@ Das Manifest enthält Metadaten über das gesamte Dokument sowie die Reihenfolge
|
||||
|
||||
```json
|
||||
{
|
||||
"format_version": 1,
|
||||
"id": "notebook-a1b2c3d4",
|
||||
"title": "Mathematik Vorlesung",
|
||||
"version": 1,
|
||||
"pages": [
|
||||
{
|
||||
"id": 1,
|
||||
"file_name": "page_1.json"
|
||||
"background": "Grid5mm"
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"file_name": "page_2.json"
|
||||
"background": "DotGrid"
|
||||
}
|
||||
],
|
||||
"page_ids": [1, 2],
|
||||
"current_page_index": 0,
|
||||
"next_page_id": 3,
|
||||
"next_stroke_id": 42,
|
||||
@@ -51,10 +54,11 @@ Das Manifest enthält Metadaten über das gesamte Dokument sowie die Reihenfolge
|
||||
```
|
||||
|
||||
### Felder:
|
||||
- `format_version`: Format-Versionsnummer (aktuell `1`, bei unversionierten Legacy-Dateien implizit `0`).
|
||||
- `id`: Eindeutige ID des Notizbuchs (z. B. UUID oder `"default"`).
|
||||
- `title`: Vom Nutzer vergebener Anzeigename des Notizbuchs.
|
||||
- `version`: Format-Versionsnummer (aktuell `1`).
|
||||
- `pages`: Geordnete Liste von Seitenreferenzen.
|
||||
- `pages`: Geordnete Liste von Seitenmetadaten (`id`, optional `background`).
|
||||
- `page_ids`: Abwärtskompatible Liste der Seiten-IDs.
|
||||
- `current_page_index`: Index der zuletzt aktiven Seite für Session-Restore.
|
||||
- `next_page_id`: Inkrementierender ID-Zähler für neue Seiten.
|
||||
- `next_stroke_id`: Globaler ID-Zähler für neue Striche.
|
||||
@@ -68,35 +72,34 @@ Jede Seite wird in einer eigenen Datei gespeichert, um Lazy-Loading und schnelle
|
||||
|
||||
```json
|
||||
{
|
||||
"format_version": 1,
|
||||
"id": 1,
|
||||
"background": "Grid5mm",
|
||||
"strokes": [
|
||||
{
|
||||
"id": 1,
|
||||
"color": [0.12, 0.12, 0.14, 1.0],
|
||||
"width_mm": 0.5,
|
||||
"samples": [
|
||||
{
|
||||
"x": 350.2,
|
||||
"y": 420.5,
|
||||
"pressure": 0.65,
|
||||
"timestamp": 12.345
|
||||
"p": 0.65,
|
||||
"t": 12.345
|
||||
},
|
||||
{
|
||||
"x": 352.1,
|
||||
"y": 425.8,
|
||||
"pressure": 0.70,
|
||||
"timestamp": 12.355
|
||||
"p": 0.70,
|
||||
"t": 12.355
|
||||
}
|
||||
],
|
||||
"recognized_shape": null
|
||||
"shape": null
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"color": [0.15, 0.35, 0.85, 1.0],
|
||||
"width_mm": 0.8,
|
||||
"samples": [],
|
||||
"recognized_shape": {
|
||||
"shape": {
|
||||
"Rectangle": {
|
||||
"top_left": { "x": 500.0, "y": 600.0 },
|
||||
"width": 300.0,
|
||||
@@ -109,14 +112,14 @@ Jede Seite wird in einer eigenen Datei gespeichert, um Lazy-Loading und schnelle
|
||||
```
|
||||
|
||||
### Felder:
|
||||
- `format_version`: Format-Versionsnummer (`1`).
|
||||
- `id`: Eindeutige ID der Seite.
|
||||
- `background`: Papiermuster (`Blank`, `Grid5mm`, `Lined`, `DotGrid`).
|
||||
- `strokes`: Liste der auf der Seite gezeichneten Striche und Vektorformen.
|
||||
- `id`: Eindeutige ID des Strichs.
|
||||
- `color`: RGBA-Farbwerte im Bereich `[0.0, 1.0]`.
|
||||
- `width_mm`: Basis-Strichbreite in Millimetern.
|
||||
- `samples`: Rohabtastpunkte entlang des Pfads (`x`, `y` in DIN-A4-Einheiten, `pressure` in `[0.0, 1.0]`, `timestamp` in Sekunden).
|
||||
- `recognized_shape`: Optionales Enum für geometrische Vektorformen:
|
||||
- `samples`: Rohabtastpunkte entlang des Pfads (`x`, `y` in DIN-A4-Einheiten, `p` Druckwert in `[0.0, 1.0]`, `t` Zeitstempel in Sekunden).
|
||||
- `shape`: Optionales Enum für geometrische Vektorformen:
|
||||
- `Line { start, end }`
|
||||
- `Rectangle { top_left, width, height }`
|
||||
- `Circle { center, radius }`
|
||||
@@ -127,8 +130,23 @@ Jede Seite wird in einer eigenen Datei gespeichert, um Lazy-Loading und schnelle
|
||||
|
||||
---
|
||||
|
||||
## 4. Atomares Schreibverfahren
|
||||
## 4. Datensicherheit & Migration
|
||||
|
||||
Um Datenverlust bei Stromausfall oder plötzlichem Prozessabbruch zu verhindern:
|
||||
1. Daten werden in `<ziel>.tmp` serialisiert und per `fsync` auf das Speichermedium geschrieben.
|
||||
### Atomares Schreiben
|
||||
1. Daten werden in `<ziel>.tmp.<pid>.<nanos>` serialisiert und per `fsync` auf das Speichermedium geschrieben.
|
||||
2. Mittels `std::fs::rename("<ziel>.tmp", "<ziel>")` wird die Zieldatei auf POSIX-Systemen atomar ersetzt.
|
||||
|
||||
### Quarantäne bei Parsefehlern
|
||||
Wird eine Seitendatei beim Laden als fehlerhaft erkannt (z. B. unvollständiges oder korruptes JSON), wird sie nicht stillschweigend gelöscht oder überschrieben, sondern nach `page_<id>.json.corrupt-<unix_ts>` verschoben. Die Ersatzseite wird als schreibgeschützt markiert.
|
||||
|
||||
### Migrations-Backups
|
||||
Beim ersten Speichern einer Datei im Legacy-Format (Version 0) wird vor dem Überschreiben ein Backup `*.v0.bak` angelegt (z. B. `manifest.json.v0.bak`, `page_1.json.v0.bak`).
|
||||
|
||||
---
|
||||
|
||||
## 5. Versionshistorie
|
||||
|
||||
| Version | Status | Beschreibung |
|
||||
|---|---|---|
|
||||
| **0** | Legacy | Unversioniertes Format. `format_version` fehlt in JSON. Implizit als Version 0 interpretiert. |
|
||||
| **1** | Aktuell | Einführung von `format_version: u32` in `manifest.json` und `page_<id>.json`. Seiten-Hintergrund im Manifest gecached (`pages: Vec<PageMeta>`). Strenges Versionsmatching (neuere Versionen werden abgelehnt). Automatische Migration mit `*.v0.bak`. |
|
||||
|
||||
+147
-3
@@ -89,9 +89,19 @@ impl StrokeData {
|
||||
}
|
||||
}
|
||||
|
||||
/// Aktuelle Dateiformat-Version von ink-notes.
|
||||
pub const CURRENT_FORMAT_VERSION: u32 = 1;
|
||||
|
||||
/// Standardwert für ältere Dateien ohne explizite Formatversion (Legacy v0).
|
||||
pub fn legacy_format_version() -> u32 {
|
||||
0
|
||||
}
|
||||
|
||||
/// Serialisierbare semantische Seitendaten.
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
pub struct PageData {
|
||||
#[serde(default = "legacy_format_version")]
|
||||
pub format_version: u32,
|
||||
pub id: u64,
|
||||
pub background: PageBackground,
|
||||
pub strokes: Vec<StrokeData>,
|
||||
@@ -100,6 +110,7 @@ pub struct PageData {
|
||||
impl From<&Page> for PageData {
|
||||
fn from(page: &Page) -> Self {
|
||||
Self {
|
||||
format_version: CURRENT_FORMAT_VERSION,
|
||||
id: page.id,
|
||||
background: page.background,
|
||||
strokes: page.strokes.iter().map(StrokeData::from).collect(),
|
||||
@@ -138,6 +149,8 @@ impl PageMeta {
|
||||
/// Metadaten des gesamten Notizbuchs (`manifest.json`).
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
pub struct NotebookManifest {
|
||||
#[serde(default = "legacy_format_version")]
|
||||
pub format_version: u32,
|
||||
pub id: String,
|
||||
pub title: String,
|
||||
pub created_at: u64,
|
||||
@@ -158,6 +171,7 @@ impl NotebookManifest {
|
||||
.unwrap_or_default()
|
||||
.as_secs();
|
||||
Self {
|
||||
format_version: CURRENT_FORMAT_VERSION,
|
||||
id: id.into(),
|
||||
title: title.into(),
|
||||
created_at: now,
|
||||
@@ -186,6 +200,7 @@ impl NotebookManifest {
|
||||
}
|
||||
|
||||
pub fn sync_from_notebook(&mut self, notebook: &Notebook) {
|
||||
self.format_version = CURRENT_FORMAT_VERSION;
|
||||
self.pages = notebook
|
||||
.pages
|
||||
.iter()
|
||||
@@ -213,6 +228,7 @@ pub enum StorageError {
|
||||
InvalidManifest(String),
|
||||
CorruptedPage(u64, String),
|
||||
PageNotFound(u64),
|
||||
UnsupportedVersion { found: u32, supported: u32 },
|
||||
}
|
||||
|
||||
impl fmt::Display for StorageError {
|
||||
@@ -225,6 +241,10 @@ impl fmt::Display for StorageError {
|
||||
write!(f, "Beschädigte Seite {id}: {msg}")
|
||||
}
|
||||
StorageError::PageNotFound(id) => write!(f, "Seite {id} nicht gefunden"),
|
||||
StorageError::UnsupportedVersion { found, supported } => write!(
|
||||
f,
|
||||
"Notizbuch wurde mit neuerer Version erstellt (Formatversion {found}, unterstützt wird bis {supported})"
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -345,6 +365,18 @@ pub fn atomic_write_json<T: Serialize>(path: &Path, value: &T) -> Result<(), Sto
|
||||
/// Speichert das Manifest atomar in `manifest.json`.
|
||||
pub fn save_manifest(notebook_dir: &Path, manifest: &NotebookManifest) -> Result<(), StorageError> {
|
||||
let path = notebook_dir.join("manifest.json");
|
||||
if path.exists() {
|
||||
let bak_path = notebook_dir.join("manifest.json.v0.bak");
|
||||
if !bak_path.exists() {
|
||||
if let Ok(content) = fs::read_to_string(&path) {
|
||||
if let Ok(old_m) = serde_json::from_str::<NotebookManifest>(&content) {
|
||||
if old_m.format_version == 0 {
|
||||
let _ = fs::copy(&path, &bak_path);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
atomic_write_json(&path, manifest)
|
||||
}
|
||||
|
||||
@@ -361,6 +393,14 @@ pub fn load_manifest(notebook_dir: &Path) -> Result<NotebookManifest, StorageErr
|
||||
let mut json_str = String::new();
|
||||
file.read_to_string(&mut json_str)?;
|
||||
let mut manifest: NotebookManifest = serde_json::from_str(&json_str)?;
|
||||
|
||||
if manifest.format_version > CURRENT_FORMAT_VERSION {
|
||||
return Err(StorageError::UnsupportedVersion {
|
||||
found: manifest.format_version,
|
||||
supported: CURRENT_FORMAT_VERSION,
|
||||
});
|
||||
}
|
||||
|
||||
if manifest.pages.is_empty() && !manifest.page_ids.is_empty() {
|
||||
manifest.pages = manifest
|
||||
.page_ids
|
||||
@@ -400,6 +440,18 @@ pub fn quarantine_corrupted_page(notebook_dir: &Path, page_id: u64) -> Result<Pa
|
||||
/// Speichert eine Seite atomar in `page_<id>.json`.
|
||||
pub fn save_page(notebook_dir: &Path, page: &PageData) -> Result<(), StorageError> {
|
||||
let path = notebook_dir.join(format!("page_{}.json", page.id));
|
||||
if path.exists() {
|
||||
let bak_path = notebook_dir.join(format!("page_{}.json.v0.bak", page.id));
|
||||
if !bak_path.exists() {
|
||||
if let Ok(content) = fs::read_to_string(&path) {
|
||||
if let Ok(old_p) = serde_json::from_str::<PageData>(&content) {
|
||||
if old_p.format_version == 0 {
|
||||
let _ = fs::copy(&path, &bak_path);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
atomic_write_json(&path, page)
|
||||
}
|
||||
|
||||
@@ -414,16 +466,25 @@ pub fn load_page(notebook_dir: &Path, page_id: u64) -> Result<PageData, StorageE
|
||||
let mut json_str = String::new();
|
||||
file.read_to_string(&mut json_str)?;
|
||||
|
||||
match serde_json::from_str::<PageData>(&json_str) {
|
||||
Ok(data) => Ok(data),
|
||||
let page_data = match serde_json::from_str::<PageData>(&json_str) {
|
||||
Ok(data) => data,
|
||||
Err(e) => {
|
||||
eprintln!(
|
||||
"[Storage-Warnung] Seite {page_id} in '{}' ist beschädigt: {e}",
|
||||
path.display()
|
||||
);
|
||||
Err(StorageError::CorruptedPage(page_id, e.to_string()))
|
||||
return Err(StorageError::CorruptedPage(page_id, e.to_string()));
|
||||
}
|
||||
};
|
||||
|
||||
if page_data.format_version > CURRENT_FORMAT_VERSION {
|
||||
return Err(StorageError::UnsupportedVersion {
|
||||
found: page_data.format_version,
|
||||
supported: CURRENT_FORMAT_VERSION,
|
||||
});
|
||||
}
|
||||
|
||||
Ok(page_data)
|
||||
}
|
||||
|
||||
/// Speichert ein gesamtes Notebook synchron auf die Festplatte (z. B. initial oder beim Flush).
|
||||
@@ -1086,4 +1147,87 @@ mod tests {
|
||||
.iter()
|
||||
.any(|name| name.starts_with("page_1.json.corrupt-")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_fixture_legacy_v0_loads_and_migrates_with_backup() {
|
||||
let temp_dir = tempfile::tempdir().expect("tempdir");
|
||||
let path = temp_dir.path();
|
||||
|
||||
let root_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
|
||||
let fixture_manifest = root_dir.join("tests/fixtures/legacy_v0_manifest.json");
|
||||
let fixture_page = root_dir.join("tests/fixtures/legacy_v0_page_1.json");
|
||||
|
||||
fs::copy(&fixture_manifest, path.join("manifest.json")).unwrap();
|
||||
fs::copy(&fixture_page, path.join("page_1.json")).unwrap();
|
||||
|
||||
let (nb, mut m) = load_notebook(path).expect("Legacy v0 notebook must load");
|
||||
assert_eq!(m.format_version, 0, "Legacy manifest has format_version 0");
|
||||
assert_eq!(nb.pages.len(), 1);
|
||||
assert_eq!(nb.pages[0].background, PageBackground::DotGrid);
|
||||
assert_eq!(nb.pages[0].strokes.len(), 1);
|
||||
|
||||
// Saving must upgrade to format_version 1 and create .v0.bak files
|
||||
save_notebook(path, &nb, &mut m).expect("Save must succeed");
|
||||
|
||||
assert!(
|
||||
path.join("manifest.json.v0.bak").exists(),
|
||||
"manifest.json.v0.bak must be created before upgrading legacy format"
|
||||
);
|
||||
assert!(
|
||||
path.join("page_1.json.v0.bak").exists(),
|
||||
"page_1.json.v0.bak must be created before upgrading legacy format"
|
||||
);
|
||||
|
||||
// Reload to verify format_version is now 1
|
||||
let (_nb_upgraded, m_upgraded) = load_notebook(path).expect("Upgraded notebook must load");
|
||||
assert_eq!(m_upgraded.format_version, 1);
|
||||
let p1_upgraded = load_page(path, 1).expect("Upgraded page must load");
|
||||
assert_eq!(p1_upgraded.format_version, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_fixture_unknown_fields_load_successfully() {
|
||||
let root_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
|
||||
let fixture_page = root_dir.join("tests/fixtures/unknown_fields_page_1.json");
|
||||
let temp_dir = tempfile::tempdir().expect("tempdir");
|
||||
let path = temp_dir.path();
|
||||
fs::copy(&fixture_page, path.join("page_1.json")).unwrap();
|
||||
|
||||
let page_data = load_page(path, 1).expect("Unknown fields must be tolerated");
|
||||
assert_eq!(page_data.background, PageBackground::Lined);
|
||||
assert_eq!(page_data.format_version, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_fixture_future_version_is_rejected() {
|
||||
let temp_dir = tempfile::tempdir().expect("tempdir");
|
||||
let path = temp_dir.path();
|
||||
|
||||
let root_dir = PathBuf::from(env!("CARGO_MANIFEST_DIR"));
|
||||
let fixture_manifest = root_dir.join("tests/fixtures/future_v99_manifest.json");
|
||||
let fixture_page = root_dir.join("tests/fixtures/future_v99_page_1.json");
|
||||
|
||||
let dest_manifest = path.join("manifest.json");
|
||||
let dest_page = path.join("page_1.json");
|
||||
fs::copy(&fixture_manifest, &dest_manifest).unwrap();
|
||||
fs::copy(&fixture_page, &dest_page).unwrap();
|
||||
|
||||
let orig_manifest_content = fs::read(&dest_manifest).unwrap();
|
||||
let orig_page_content = fs::read(&dest_page).unwrap();
|
||||
|
||||
// Loading must fail with UnsupportedVersion
|
||||
let load_res = load_notebook(path);
|
||||
assert!(load_res.is_err(), "Future version must be rejected");
|
||||
match load_res.unwrap_err() {
|
||||
StorageError::UnsupportedVersion { found, supported } => {
|
||||
assert_eq!(found, 99);
|
||||
assert_eq!(supported, CURRENT_FORMAT_VERSION);
|
||||
}
|
||||
other => panic!("Expected UnsupportedVersion, got: {other:?}"),
|
||||
}
|
||||
|
||||
// Files on disk must remain completely untouched
|
||||
assert_eq!(fs::read(&dest_manifest).unwrap(), orig_manifest_content);
|
||||
assert_eq!(fs::read(&dest_page).unwrap(), orig_page_content);
|
||||
}
|
||||
}
|
||||
|
||||
+14
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"format_version": 99,
|
||||
"id": "future_notebook",
|
||||
"title": "Zukunfts-Notizbuch",
|
||||
"created_at": 1800000000,
|
||||
"modified_at": 1800001000,
|
||||
"current_page_index": 0,
|
||||
"pages": [
|
||||
{ "id": 1, "background": "Grid5mm" }
|
||||
],
|
||||
"page_ids": [1],
|
||||
"next_stroke_id": 10,
|
||||
"next_page_id": 2
|
||||
}
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"format_version": 99,
|
||||
"id": 1,
|
||||
"background": "Blank",
|
||||
"strokes": []
|
||||
}
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"id": "legacy_notebook",
|
||||
"title": "Legacy Notizbuch v0",
|
||||
"created_at": 1700000000,
|
||||
"modified_at": 1700001000,
|
||||
"current_page_index": 0,
|
||||
"page_ids": [1],
|
||||
"next_stroke_id": 5,
|
||||
"next_page_id": 2
|
||||
}
|
||||
Vendored
+14
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"id": 1,
|
||||
"background": "DotGrid",
|
||||
"strokes": [
|
||||
{
|
||||
"id": 1,
|
||||
"color": [0.1, 0.2, 0.3, 1.0],
|
||||
"samples": [
|
||||
{ "x": 100.0, "y": 200.0, "p": 0.5, "t": 1.0 },
|
||||
{ "x": 110.0, "y": 210.0, "p": 0.6, "t": 1.1 }
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"format_version": 1,
|
||||
"id": 1,
|
||||
"background": "Lined",
|
||||
"strokes": [],
|
||||
"unknown_experimental_feature": {
|
||||
"nested_data": [1, 2, 3],
|
||||
"author": "Antigravity"
|
||||
},
|
||||
"random_future_flag": true
|
||||
}
|
||||
Reference in New Issue
Block a user