From 683944bafc1a07266fe048dce2bc0a151597a85f Mon Sep 17 00:00:00 2001 From: Ian Jackson Date: Tue, 25 Jan 2011 15:43:21 +0000 Subject: [PATCH] notes: new text format --- yarrg/notes | 144 ++++++++++++++++++++-------------------------------- 1 file changed, 55 insertions(+), 89 deletions(-) diff --git a/yarrg/notes b/yarrg/notes index 8895c71..e1e9a14 100644 --- a/yarrg/notes +++ b/yarrg/notes @@ -52,44 +52,71 @@ very few price changes, mostly qty changes ==================== files - ARCHIVE-%{ocean}s-lock.par never removed - ARCHIVE-%{ocean}s-main.par updated by rename - ARCHIVE-%{ocean}s-auxil.par appended/renamed, len in main - ARCHIVE-%{ocean}s-log-%{isleid}s.par appended/renamed, len in main - ARCHIVE-%{ocean}s-old-%{isleid}s-%4d.par.gz created, count in main + ARCHIVE-%{ocean}s-lock.par + never removed + + ARCHIVE-%{ocean}s-main.par + updated using rename + contains: magic, main, commod*, diff* + + ARCHIVE-%{ocean}s-auxil.par + appended or renamed; length is recorded in main + contains: magic, auxil*, garbage (after recorded len) + + ARCHIVE-%{ocean}s-log-%{islandid}s.par + appended or renamed; length is recorded in main + refers to commods and auxils from main and auxil + contains: magic, (diff, difflen)*, garbage (after recorded len) + + ARCHIVE-%{ocean}s-arc-%{islandid}s-%4d.par.gz + created in place; valid only if main mentions it + refers to commods and auxils in this file only + however commods must be the same as those in main + contains gzipped data for: + magic, commod*, auxil*, diff* others files is always updated before main so lockfree readers should open main, then other files -format is a series of lines +format of each file except lock is a series of lines - all !yarrg-archive [...] magic, 1st line + magic !yarrg-archive [...] magic, 1st line - main !stalls [...] - main !island \ + main !island \ [...] " - main = + main !auxil [...] + + commod = auxil =: auxil =&\n + the metadata has these keywords removed: ocean, island, timestamp - log/old & + diff & applies to following data - log/old [:] delete offer - log/old [:]@ adjust price - log/old [:][@]- adjust qty - log/old [:][@]+ maybe price + diff [:] delete offer + diff [:]@ adjust price + diff [:][@]- adjust qty + diff [:][@]+ maybe price In each case commodiddelta is the number to add to the previous commodid to get the new one; or for a new stallid, the actual commodid (starting from zero). -ve numbers cannot be represented so they have to be in order. - log ^ + difflen ^ applies to previous data includes 6-byte len of one ^... line +all numbers (including ids, byte counts, numbers of files) (but not +including the arc file number %d in the arc file name) are in base64 +MS digit first using the following digits: + 0 1 2 3 4 5 6 7 8 9 + a b c d e f g h i j k l m n o p q r s t u v w x y z + A B C D E F G H I J K L M N O P Q R S T U V W X Y Z + _ . + log file is a series of diffs most recent last; each diff is metadataid×tamp, zero or more commods, ^uploadlenbytes the commodid @@ -97,76 +124,15 @@ log file is a series of diffs most recent last; each diff is for a log file, there may be some trailing garbage not referred to in main file (see "length of log file") - for a z file, the file length is definitive and the last - entry is always valid if the file is referred to in the main - file, but any z file not mentioned in the main file is - garbage - -format for a diff - Each diff records a change "going backwards", ie you apply the - diff to a more recent state to get an earlier state; the metadata - corresponds to the earlier state; diffs at the physical end, ie - logical start, of the file, contain information without previous - context (ie start from empty, no offers) - - diff format version Plain uint8 constant 0x01 - - timestamp delta - (amount by which this timestamp - is later than the previous upload in this file, - or later than 0 if there is no previous pload in this file) - Plain vuint time_t - - for each payload stream, ie: - for Meta, Stall, Commod, Price, Qty: - in an uncompressed diff: - uncompressed data Plain some number of bytes - the uncompressed data is in the order shown below - in a compressed diff: - compressed length Plain vuint - compressed data Plain that number of bytes - the compressed data for each stream forms a continuous - compression stream within each file, starting with - the last diff in the file and then running backwards - - diff length (reverse pointer) Frame uint32 - includes length of exactly the - data sections marked "Plain" - -format inside the payload streams - uncompressed streams, literally in this order - compressed streams: read each substream in order, but ordering - between substreams with a diff is semantic but not physical - - metadata excluding - ocean, island, timestamp Meta uint16 metadata length - Meta uint8*length metadata - - for each stall - - stall name Stall uint16 name length - Stall uint8*length name - - for each commod which has changed price - including ones which have been added - - commodid Commod uint16 commodity id - - buy price delta Price uint16 - sell price delta Price uint16 - in case of added offers, previous price - is taken to be best price from previous upload - at this island, or 0 if previously no offers - - sentinel commodid Commod uint16 constant 0x0000 - - for each commod which has changed qty - including ones which have been added or removed - - commodid Commod uint16 commodity id - buy qty delta Qty uint16 - sell qty delta Qty uint16 - - sentinel commodid Commod uint16 constant 0x0000 - - sentinel stall name Stall uint16 constant 0x0000 + for an arc file, the file length is definitive and the last + entry is always valid if the file is referred to in the main + file, but any arc file not mentioned in the main file is + garbage + +semantics of a diff + Each diff records a change "going backwards", ie the diff describes a + previous state in terms of a more recent state and you apply the diff + to a more recent state to get the earlier state described by the + diff; the metadata corresponds to the earlier state; diffs at the + physical end, ie logical start, of the file, contain information + without previous context (ie start from empty, no offers) -- 2.30.2