This tool can:
- screenscrape the commodities trading screen
- produce the results as a tab separated values file
- - **TODO** upload the results to PCTB
+ - upload the results to PCTB
To run it, change to this directory, type `make', and then:
+ ./ypp-commodities --upload
+to upload to the PCTB server. Currently we use pctb.ilk.org,
+the testing instance of the PCTB database, pending approval from the
+operators of the main server.
+
+Or, for example, for a tab-separated values dump:
./ypp-commodities --tsv >commods.tsv
While it is capturing the screenshots, do not move the mouse or use
--screenshot-file F Store or read screenshots in F rather than #pages#.pnm
--window-id ID Specified X window is the YPP client - do not search
--edit-charset Enable character set editing. See README.dictionary.
+ --no-edit-charset Do not edit charset even if #local-char*#.txt exists.
+ --find-island Find and print the ocean and island. Suppresses OCR
+ and output unless used with result processing option.
+ --test-servers Set default servers to be the test servers, not
+ the real live ones (doesn't affect explicit settings).
-Controlling what happens to the results:
+Controlling what happens to the results - only one at a time:
--upload (default) Upload to the PCTB server
--tsv Print data as clean tab-separated-values file
--raw-tsv Dump the raw (not deduped, unsorted) OCR'd data
Please do not use options marked * with --upload. See README.privacy.
Options to override which servers we talk to:
- --pctb-url HOST|URL Talk to the PCTB server at HOST or URL.
- --dict-submit-url URL Submit dictionary entries with HTTP POST under URL.
- --dict-update-url URL Fetch updated master dictionary with rsync from URL.
-
-
-Files we use and update
------------------------
-
-The program reads and writes the following files:
-
- * #pages#.pnm
-
- Contains one or more images (as raw ppms, end-to-end) which are the
- screenshots taken in the last run. This is (over)written whenever
- we take screenshots from the YPP client. You can reprocess an
- existing set of screenshots with the --same (aka --analyse-only)
- option; in that case we just read the screenshots file.
-
- You can specify a different file with --screenshot-file.
-
- If you want to display the contents of this file, `display' can do
- it. Don't try `display vid:#pages#.pnm' as this will consume
- truly stupendous quantities of RAM - it wedged my laptop.
-
- * charset-15.txt
-
- Character set dictionary. For the semantics of the contents of this
- file see README.charset. There is not currently any accurate
- documentation of this dictionary format.
-
- If you delete this file you'll have to re-enter a lot of glyph data
- (and probably get it wrong and make the program misrecognise
- things). If you want to undo any mistakes you may have made
- answering OCR questions you can safely revert this to the version
- I've supplied.
-
- * #commodmap#.tsv
-
- Map from commodity names to the numbers required by the PCTB
- server. This is fetched and updated automatically as necessary.
- It can safely be deleted as it will then be refetched.
-
- * <file>.new
-
- When any of these tools overwrite one of the persistent dictionary
- files, they temporarily write to <file>.new.
-
-These files are all in the current working directory. There is not
-yet any feature to have them be somewhere else. The helper programs
- yppsc-ocr-resolver
- yppsc-commod-processor
-must (currently) also be in the current directory.
-
-Future versions may have more helpers and more data files.
+ --pctb-server HOST|URL Talk to the PCTB server at HOST or URL.
+ --dict-submit-url URL Submit dictionary entries with HTTP POST under URL.
+ --dict-update-from SRC Fetch updated master dictionary with rsync from SRC.
+Or set the environment variables YPPSC_PCTB{_PCTB, _DICT_UPDATE, _DICT_SUBMIT}
Installation requirements
- pnm command line utilities for image manipulation netpbm
- X11 libraries, including dev files for building libx11-dev
- XTEST library, including dev files for building libxtst-dev
+ - Perl-compatible regexp library, including dev files libpcre3-dev
- Tk interpreter /usr/bin/wish tk8.4
- Perl module XML::Parser libxml-parser-perl
- Perl module JSON::Parser libjson-perl
- Perl interpreter and basic modules perl (usu.installed)
On other Linux distros the packages may have different names, but
-these should be roughly right for Debian and its derivatives.
+these should be roughly right for Debian and its derivatives. You can
+install them with this rune:
+ sudo apt-get install build-essential libnetpbm10-dev netpbm libx11-dev libxtst-dev libpcre3-dev tk8.4 libxml-parser-perl libjson-perl
+
+
+The supplied helper programs
+ dictionary-manager
+ commod-results-processor
+ database-info-fetch
+must (currently) also be in the current working directory when you run
+the main ypp-commodities program.
+
+The data files (see README.files) are also left in the current working
+directory. There is notyet any feature to have the data files and
+helpers be somewhere else.
Reporting problems
~yarrgweb / ypp-sc-tools.db-test.git / blobdiff