+YARRG - Yet Another Revenue Research Gatherer
+=============================================
+
Overview
--------
This tool can:
- screenscrape the commodities trading screen
- produce the results as a tab separated values file
- - upload the results to PCTB
+ - upload the results to the YARRG and PCTB servers
To build, install the dependencies, cd to this directory and type `make'.
-To screenscrape and upload to the PCTB server, select `trade
+To screenscrape and upload to both servers, select `trade
commodities' from the hold of a vessel or building, and run:
./ypp-commodities
-Currently we use pctb.ilk.org, the testing instance of the PCTB
-database, pending approval from the operators of the main server.
+Currently we upload to the dedicated yarrg server yarrg.chiark.net,
+and also to pctb.ilk.org (the testing instance of the PCTB database,
+pending approval from the operators of the main server).
Or, for example, for a tab-separated values dump:
./ypp-commodities --tsv >commods.tsv
the real live ones (doesn't affect explicit settings).
Controlling what happens to the results - only one at a time:
- --upload (default) Upload to the PCTB server
+ --upload (default) Upload to the YARRG and PCTB servers
--tsv Print data as clean tab-separated-values file
--raw-tsv Dump the raw (not deduped, unsorted) OCR'd data
--best-prices Print best buy and sell price for each commodity
Please do not use options marked * with --upload. See README.privacy.
Options to override which servers we talk to:
- --pctb-server HOST|URL Talk to the PCTB server at HOST or URL.
+ --yarrg-server HOST|URL When uploading to YARRG, use HOST or URL.
+ --pctb-server HOST|URL Talk to the PCTB database at HOST or URL.
--dict-submit-url URL Submit dictionary entries with HTTP POST under URL.
--dict-update-from SRC Fetch updated master dictionary with rsync from SRC.
-Or set the environment variables YPPSC_PCTB{_PCTB, _DICT_UPDATE, _DICT_SUBMIT}
+Or set the environment variables YPPSC_YARRG{_PCTB, _DICT_UPDATE, _DICT_SUBMIT}
Installation requirements
Privacy
-------
-The main purpose of this program is to connect to the PCTB server and
-upload data. It will do that if you run it with --upload.
+The main purpose of this program is to connect to the YARRG and PCTB
+servers and upload data. It will do that if you run it with --upload.
This program will also, by default, talk to the dictionary server I
have set up: to download updated image dictionaries, and to upload new
-dictionary entries which you create with the PCTB client dictionary
+dictionary entries which you create with the yarrg client dictionary
GUI. This feature is mentioned in and controllable in the GUI itself,
so it won't happen without you knowing about it.
Authorship and copyright
------------------------
-The PCTB server itself is a project of Joel Lord and various others.
+The PCTB server is a project of Joel Lord and various others.
+
+The YARRG server was inspired by PCTB and is a project of the crew
+Special Circumstances (on the Midnight Ocean) and of the Sinister
+Greenend Organisation.
-However, this screenscraper and upload client was written entirely
-from scratch by me, Ian Jackson. It is part of ypp-sc-tools, a set of
-third-party tools for assisting players of Yohoho Puzzle Pirates.
+This yarrg screenscraper and upload client was written entirely from
+scratch by me, Ian Jackson, with assistance from Stephen Early. It is
+part of ypp-sc-tools, a set of third-party tools for assisting players
+of Yohoho Puzzle Pirates.
ypp-sc-tools is
Copyright (C) 2009 Ian Jackson <ijackson@chiark.greenend.org.uk>
+Copyright (C) 2009 Stephen Early
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
The character and UI images copied from the YPP client, and submitted
-to stored and shared by the YPP SC PCTB dictionary server, are those
+to stored and shared by the YPP SC YARRG dictionary server, are those
provided by Three Rings as part of the YPP client and by your Java
installation. I regard the current use of these images in this way as
Fair Dealing (in the UK) or Fair Use (in the USA).
These images do not form part of the ypp-sc-tools distribution,
-although the ypp-sc-tools pctb client does download them automatically
+although the ypp-sc-tools yarrg client does download them automatically
from my dictionary server when run in the most ordinary way.
example, the recognised data is wrong), you should delete the file
_local-char*.txt, which contains your local updates. It will then
only use the centrally provided (and vetted) master file (which is
-automatically updated when you run the PCTB client, by default).
+automatically updated when you run the yarrg client, by default).
It is also possible to have the OCR system reject particular strings.
If you put a regexp in _local-reject.txt, any OCR result which
* _master-newcommods.txt _local-newcommods.txt
Dictionary of newly introduced commodities. When a new commodity
- appears in Puzzle Pirates, the PCTB server operators need to add it
- to their database for us to be able to upload data about it.
+ appears in Puzzle Pirates, the YARRG and PCTB server operators each
+ need to add it to their database for us to be able to upload data
+ about it.
It can sometimes take a few days to do this. In the meantime, it
is possible to upload partial data - data just omitting that
commodity. This is controlled by these files: they list
- commodities which should be automatically ignored if the PCTB
- server doesn't know about them. The master file is downloaded and
- updated automatically from my server. You may create the local
- file yourself. The format is simple: one commodity per line.
+ commodities which should be automatically ignored if the server
+ doesn't know about them. The master file is downloaded and updated
+ automatically from my server. You may create the local file
+ yourself. The format is simple: one commodity per line.
Unrecognised commodities can also be due to OCR failure so
double-check what you're doing before overriding the uploader by
documentation of this dictionary format.
_master-*.txt contain the centrally defined and approved data.
- They are downloaded automatically from the SC PCTB server and
+ They are downloaded automatically from the SC YARRG server and
updated each run. You can safely delete these files, if everything
is online, if you want to fetch a fresh copy.
* _upload-1.html _upload-2.html _upload-3.html
- We screenscrape the pages from the PCTB upload server. The actual
- HTML returned from the upload server is left in these dropping
- files for debugging etc.
+ We HTML-screenscrape the pages from the PCTB upload server. The
+ actual HTML returned from the upload server is left in these
+ dropping files for debugging etc.
* _<file>.tmp
Communications with servers - and your privacy
==============================================
-The Special Circumstances YPP PCTB client talks to two different
+The Special Circumstances YPP yarrg client talks to several different
servers for different purposes.
(The information below is true if you invoke the system using the main
behaviours of the helpers, you'll have to read the source. Sorry.)
-1. PCTB server
-==============
+1. YARRG and PCTB servers
+=========================
-This records everyone's uploads of commodity prices and allows you to
-conveniently search for good trade routes using the PCTB website.
+These servers records everyone's uploads of commodity prices and
+allows you to search for routes using the PCTB and YARRG websites.
+(*NB* currently the YARRG website is still under development.)
-We upload to the PCTB server if you select the --upload option, and
-not otherwise.
+The YARRG data upload server may also distributes the uploaded data to
+other searching site operators for inclusion in their databases; your
+IP address is not passed on to those other operators.
+
+We upload to the PCTB and YARRG servers if you select one of the
+--upload options, and not otherwise - but not that this is the
+default.
We also query the PCTB server to determine possible island names, if
we don't recognise the island and want to ask you about it. It is not
won't happen.
-2. YPP SC PCTB client dictionary server
-=======================================
+2. YPP SC YARRG dictionary server
+=================================
This server maintains the master copies of various databases which are
-used to assist the OCR process. Collectively I call these
-`dictionaries'. There is a dictionary of character and island name
-images, of new commodity names, and of notable known OCR
-misrecognitions. These are used for the commodity screen OCR, for
-determining your island name, and for checking whether commodities not
-found on the PCTB server are real.
-
-By default, we ask the server for appropriate updated dictionaries
-every time we run; this is done with the rsync protocol (indeed, by
-invoking rsync). You can disable this with --dict-local-only.
+used to assist the OCR and upload process. Collectively I call these
+`dictionaries'. There are dictionaries of character and island name
+images, of (existing and new) commodity names, of notable known OCR
+misrecognitions, and of islands and archipelagoes. These are used for
+the commodity screen OCR, for determining your island name, and for
+checking whether commodities missing on the servers are real.
+
+By default, we ask the YARRG disctionary server for appropriate
+updated dictionaries every time we run; this is done with the rsync
+protocol (indeed, by invoking rsync). You can disable this with
+--dict-local-only.
If we find a screen display we don't understand, we will ask you about
it by popping up a window which allows you to select the island (or
(for example, to let you know if you have made a mistake, or to thank
you for your contributions).
-However, if you prefer to be anonymous, you can tell your PCTB client
+However, if you prefer to be anonymous, you can tell your yarrg client
not to mention your pirate name (in the GUI, or with the --dict-anon
option). In this case I won't see your ocean or your pirate name,
although of course the actual images in your submissions may reveal
increase your privacy setting, use the --dict-* privacy options.
See the table of options in the README for details.
-Having said all that, please do not upload data to the PCTB server
-without also participating in dictionary sharing. If you don't update
-your dictionary, your parses may be wrong and thus the data you upload
-to PCTB may be wrong. If you don't submit your dictionary entries,
-any mistakes you make will remain uncorrected.
+Having said all that, please do not upload data to the YARRG and PCTB
+servers without also participating in dictionary sharing. If you
+don't update your dictionary, your parses may be wrong and thus the
+data you uploads may be wrong. If you don't submit your dictionary
+entries, any mistakes you make will remain uncorrected.
Records kept
------------
-I keep a permanent log of all the submissions, including date, time,
-submitting pirate or IP address, and YPP SC PCTB client version.
-This is so that I have enough information to go back and fix the
-dictionary if anything goes badly wrong (for example, if a particular
-client is broken).
+I keep a permanent log of all the dictionary submissions, including
+date, time, submitting pirate or IP address, and YPP SC yarrg client
+version. This is so that I have enough information to go back and fix
+the dictionary if anything goes badly wrong (for example, if a
+particular client is broken).
+
+I keep a log of the data uploads including date, time, submitting IP
+address, YPP SC yarrg client version but NOT including your pirate
+name. This information (except your IP address) is passed on to any
+third parties who've asked to get copies of data updates.
My rsync server (file server) also records your IP address when your
client fetches new master dictionaries; those logs are used only for
~yarrgweb / ypp-sc-tools.web-live.git / commitdiff