notes: new text format

diff --git a/yarrg/notes b/yarrg/notes
index 8895c7189430f6216e837c29c06803c3c473fbd0..e1e9a14abba4aac942e84b7e1a689cd05f479ba7 100644 (file)
--- a/yarrg/notes
+++ b/yarrg/notes
@@ -52,44 +52,71 @@ very few price changes, mostly qty changes
 ====================
 
 files
-  ARCHIVE-%{ocean}s-lock.par                   never removed
-  ARCHIVE-%{ocean}s-main.par                   updated by rename
-  ARCHIVE-%{ocean}s-auxil.par                  appended/renamed, len in main
-  ARCHIVE-%{ocean}s-log-%{isleid}s.par         appended/renamed, len in main
-  ARCHIVE-%{ocean}s-old-%{isleid}s-%4d.par.gz  created, count in main
+  ARCHIVE-%{ocean}s-lock.par
+       never removed
+
+  ARCHIVE-%{ocean}s-main.par
+       updated using rename
+       contains: magic, main, commod*, diff*
+
+  ARCHIVE-%{ocean}s-auxil.par
+       appended or renamed; length is recorded in main
+       contains: magic, auxil*, garbage (after recorded len)
+
+  ARCHIVE-%{ocean}s-log-%{islandid}s.par
+       appended or renamed; length is recorded in main
+       refers to commods and auxils from main and auxil
+       contains: magic, (diff, difflen)*, garbage (after recorded len)
+
+  ARCHIVE-%{ocean}s-arc-%{islandid}s-%4d.par.gz
+       created in place; valid only if main mentions it
+       refers to commods and auxils in this file only
+       however commods must be the same as those in main
+       contains gzipped data for:
+               magic, commod*, auxil*, diff*
 
  others files is always updated before main
  so lockfree readers should open main, then other files
 
-format is a series of lines
+format of each file except lock is a series of lines
 
- all   !yarrg-archive [...]                            magic, 1st line
+ magic !yarrg-archive [...]                            magic, 1st line
 
- main  !stalls <stallslenbytes> [...]
- main  !island <islandid> <islandloglenbytes> <islandoldfiles> \
+ main  !island <islandid> <islandloglenbytes> <islandarcfiles> \
                [...] "<islandname>
- main  <commodid>= <commodname>
+ main  !auxil <auxillenbytes> [...]
+
+ commod        <commodid>= <commodname>
 
  auxil <stallid>=:<stallname>
  auxil <metadataid>=&<metadatalenbytes>\n<metadata>
+       the metadata has these keywords removed: ocean, island, timestamp
 
- log/old <metadataid>&<timestamp>
+ diff  <metadataid>&<timestamp>
        applies to following data
 
- log/old [<stallid>:]<commodiddelta>                            delete offer
- log/old [<stallid>:]<commodiddelta>@<newprice>                         adjust price
- log/old [<stallid>:]<commodiddelta>[@<newprice>]-<qtyreduced>  adjust qty
- log/old [<stallid>:]<commodiddelta>[@<newprice>]+<qtyincreased>  maybe price
+ diff  [<stallid>:]<commodiddelta>                              delete offer
+ diff  [<stallid>:]<commodiddelta>@<newprice>                   adjust price
+ diff  [<stallid>:]<commodiddelta>[@<newprice>]-<qtyreduced>    adjust qty
+ diff  [<stallid>:]<commodiddelta>[@<newprice>]+<qtyincreased>  maybe price
        In each case commodiddelta is the number to add to
        the previous commodid to get the new one; or for a new
        stallid, the actual commodid (starting from zero).
        -ve numbers cannot be represented so they have to be in
        order.
 
- log    ^<uploadlenbytes-padded-to-4-base62-digits>
+ difflen ^<uploadlenbytes-padded-to-4-"digits">
                        applies to previous data
                        includes 6-byte len of one ^... line
 
+all numbers (including ids, byte counts, numbers of files) (but not
+including the arc file number %d in the arc file name) are in base64
+MS digit first using the following digits:
+ 0 1 2 3 4 5 6 7 8 9
+ a b c d e f g h i j k l m n o p q r s t u v w x y z
+ A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
+ _ .
+
 log file is a series of diffs most recent last; each diff is
   metadataid&timestamp, zero or more commods, ^uploadlenbytes
   the commodid
@@ -97,76 +124,15 @@ log file is a series of diffs most recent last; each diff is
  for a log file, there may be some trailing garbage
  not referred to in main file (see "length of log file")
 
-       for a z file, the file length is definitive and the last
-        entry is always valid if the file is referred to in the main
-        file, but any z file not mentioned in the main file is
-        garbage
-
-format for a diff
- Each diff records a change "going backwards", ie you apply the
- diff to a more recent state to get an earlier state; the metadata
- corresponds to the earlier state; diffs at the physical end, ie
- logical start, of the file, contain information without previous
- context (ie start from empty, no offers)
-
-       diff format version                     Plain   uint8 constant 0x01
-
-       timestamp delta
-        (amount by which this timestamp
-        is later than the previous upload in this file,
-        or later than 0 if there is no previous pload in this file)
-                                               Plain   vuint time_t
-
-       for each payload stream, ie:
-       for Meta, Stall, Commod, Price, Qty:
-           in an uncompressed diff:
-               uncompressed data               Plain   some number of bytes
-                the uncompressed data is in the order shown below
-           in a compressed diff:
-               compressed length               Plain   vuint
-               compressed data                 Plain   that number of bytes
-                the compressed data for each stream forms a continuous
-                compression stream within each file, starting with
-                the last diff in the file and then running backwards
-
-       diff length (reverse pointer)           Frame   uint32
-        includes length of exactly the
-        data sections marked "Plain"
-
-format inside the payload streams
- uncompressed streams, literally in this order
- compressed streams: read each substream in order, but ordering
-  between substreams with a diff is semantic but not physical
-
-       metadata excluding
-        ocean, island, timestamp               Meta    uint16 metadata length
-                                               Meta    uint8*length metadata
-
-       for each stall
-
-               stall name                      Stall   uint16 name length
-                                               Stall   uint8*length name
-
-               for each commod which has changed price
-                including ones which have been added
-
-                       commodid                Commod  uint16 commodity id
-
-                       buy price delta         Price   uint16
-                       sell price delta        Price   uint16
-                         in case of added offers, previous price
-                         is taken to be best price from previous upload
-                         at this island, or 0 if previously no offers
-
-               sentinel commodid               Commod  uint16 constant 0x0000
-
-               for each commod which has changed qty
-                including ones which have been added or removed
-
-                       commodid                Commod  uint16 commodity id
-                       buy qty delta           Qty     uint16
-                       sell qty delta          Qty     uint16
-
-               sentinel commodid               Commod  uint16 constant 0x0000
-
-       sentinel stall name                     Stall   uint16 constant 0x0000
+ for an arc file, the file length is definitive and the last
+ entry is always valid if the file is referred to in the main
+ file, but any arc file not mentioned in the main file is
+ garbage
+
+semantics of a diff
+ Each diff records a change "going backwards", ie the diff describes a
+ previous state in terms of a more recent state and you apply the diff
+ to a more recent state to get the earlier state described by the
+ diff; the metadata corresponds to the earlier state; diffs at the
+ physical end, ie logical start, of the file, contain information
+ without previous context (ie start from empty, no offers)