Leo's Geocaching-Site

Diese Site durchsuchen
Paperless Caching‎ > ‎PqDL‎ > ‎

PqDL (EN)

(added on request)

I am posting new updates on blog.leoluk.de at the moment. The most recent release at the time of this writing is 0.3.3, the GC.com update bug has been fixed.
Das ist die englische Projektseite. Um zur deutschen Seite zu kommen, hier klicken.

Inhalt

  1. 1 Screenshot
  2. 2 Manual
  3. 3 Usage
  4. 4 Download
    1. 4.1 Crashes while parsing the site
  5. 5 Changelog
    1. 5.1 04.06.2010 - Version 0.1.0
    2. 5.2 06.06.2010 - Version 0.2.0
    3. 5.3 06.06.2010 - Version 0.2.1
    4. 5.4 12.06.2010 - Version 0.2.2
    5. 5.5 12.06.2010 - Version 0.2.3
    6. 5.6 12.06.2010 - Version 0.2.4
    7. 5.7 18.06.2010 - Version 0.3.0
  6. 6 Sourcecode (for Linux/Mac/etc.)
  7. 7 Beta versions
  8. 8 Bugs and Feature Requests


Click here to visit the Sourceforge project page.

Disclaimer und GC.com ToU

The ToU on geocaching.com probably don't allow the use of this script.

I published it anyway because it doesn't create more traffic and doesn't allow non-PM members to get information they should pay for.

Of course, use this on your own risk and don't abuse it.
PqDL is considered as a flexible yet useful tool which can help to download Pocket Queries that have more than 500 caches from geocaching.com. Pocket Queries that contain more than 500 caches won't be sent per mail. Therefore, you need to do it by hand or with this script.

If you have questions, feel free to contact me per chat:









Screenshot



Manual

This is a command-line tool, so it can be used in conjunction with GSAK. You could make it download the files to the temp folder of the GSAK macro "pqloader" which will unzip and import them. When using -j, the PQs that are already downloaded won't be downloaded again, even if they get removed.

Command-line options

-> PQdl v0.3.1-stable by leoluk. Updates and help on www.leoluk.de/paperless-caching/pqdl

Usage: pqdl.py [-h] -u USERNAME -p PASSWORD [-o OUTPUTDIR] [options] [pq_1 pq_2 ...]

PQdl is a tool that can download Pocket Queries from geocaching.com.  Pocket
Queries that contain more than 500 caches won't be sent per mail, so you  need
to do it by hand or with this script. This script is written by leoluk. Please
look at www.leoluk.de/paperless-caching/pqdl for updates.

Options:
  --version             show program's version number and exit
  -h, --help            show this help message and exit
  -u USERNAME, --username=USERNAME
                        Username on GC.com (use parentheses if it contains
                        spaces)
  -p PASSWORD, --password=PASSWORD
                        Password on GC.com (use parentheses if it contains
                        spaces), you will be asked if you don't specify it
                        (you can omit this!)
  -o OUTPUTDIR, --outputdir=OUTPUTDIR
                        Output directory for downloaded files (will be created
                        if it doesn't exists yet), will be set as default for
                        other file parameters, sets the working dir [default:
                        D:\Eigene Dateien\Python\pqdl]
  -r, --remove          Remove downloaded files from GC.com. WARNING: This
                        deletes the files ONLINE! WARNING: This is broken from
                        time to time, thanks go to Groundspeak!
  -n, --nospecial       Ignore special Pocket Queries that can't be removed
                        like My Finds.
  -s, --singlefile      Overwrite existing files. When using this option,
                        there won't be any timestamps added! (so just one file
                        for every PQ in your DL folder), applies to unzip too
  -e, --delay           Random delays between the requests
  -l, --list            Do not download anything, just list the files. Best to
                        be used with -d.
  --myfinds             Trigger a My Finds Pocket Query if possible (you'll
                        most likely need to run this program again if the PQ
                        is not generated fast enough, so consider using
                        --myfinds with -l)

  Arguments:
    Pass the names of the Pocket Queries you want to download as
    parameters (pq_1 pq_2 ...). (case sensitive!) If none given, it will
    try to download all of them. You can exlude PQs by adding # on the
    beginning of the name. You need to specify the 'friendly name', the
    name, the date, the cache count or the ID of a PQ. You can use UNIX-
    style wildcards (*, ?, [x], [!x]). Please run with -d -l to get the
    friendly name or other parameters.

  ZIP options:
    PqDL supports unzipping the Pocket Queries. They will be renamed
    automatically after unzipping by this pattern: Name-of-
    PQ_1234567_06-12-2010[_waypoints].gpx (-s will be used). Note: if you
    want to your PQs with GSAK or pqloader, there's no need to unzip them!

    -z, --unzip         Unzips and removes the downloaded ZIP files.
    --keepzip           Do not remove unzipped files. (to be used with -z)

  Debug options:
    They are lots of debug options. You should always use -d if the
    program doesn not exactly does what it's supposed to do, they are lots
    of interesting debug outputs. The other debug options are only needed
    for debugging special problems with the parser. PLEASE ALWAYS USE -d
    IF YOU SEND ME A BUG REPORT!

    -d, --debug         Debug output (RECOMMENDED)
    -t, --httpdebug     HTTP header debug output, used for debugging fake
                        requests
    --httpremovedebug   HTTP 'remove PQ' debug output, used when -r doesn't
                        works
    --ignoreerrors      Ignore version errors
    --httpmaindebug     HTTP 'getPQ' debug output, used for debugging the main
                        parser that gets the PQ site table
    --ctl=CTL           Remove-CTL value, used for debugging very special
                        problems with -r (default: search)

  Journal and map options:
    These are special options that will allow PqDL to remember which PQs
    have already been downloaded. This is based on the PQ latest
    generation date, if the PQ gets generated again, it will be
    downloaded. The journal file is an .ini file (by default
    filestate.txt) that can be used for the mappings too. The section for
    this feature is [Log].

    -j, --journal       Create a download journal file. Files downloaded while
                        using -j there won't be downloaded again (requires -j
                        or --usejournal)
    --usejournal        Like -j, but in read-only mode, it won't add new PQs
                        or pqloader mappings to the journal (-j or this one!)
    --resetjournal      Reset the log section of the journal
    --journalfile=JOURNALFILE
                        Filename of journal file [default: filestate.txt]

  GSAK/pqloader file mappings options:
    This is a feature made for those who use PqDL in conjunction with
    pqloader. pqloader will take the first word in a PQs file name to
    decide in which database the PQs will be saved. This feature allows
    you to add this prefix automatically after downloading the PQs, so you
    don't longer need to rename your PQs online! This feature will use an
    .ini file like -j (this can be the same one, the default is
    filestate.txt too). In order to use this, you need to add a new
    section [Map] to the .ini file and mappings like My-PQ-Name=PQ-Prefix
    (one per line). You can use the name, friendlyname, date or ID, but no
    wildcards yet.

    -m, --mappings      Assign a GSAK Database for pqloader to every PQ,
                        requires journal
    --mapfile           File that contains the mapping section, default is the
                        journal file. [default: filestate.txt, or the custom
                        journal file if set]. For usage examples look at the
                        project site.
    --sep               Seperator for pqloader [default: ' ']

This tool probably violates the Terms of Service by Groundspeak.  Please don't
abuse it. If any argument (username, password, PQ names, ...) contains spaces,
put it into parantheses.

Usage

The usage of this tool should be self-explanatory. This section only lists a few options, read the command-line options above if you want to see all possibilities!

If you want to see what it does exactly of if the result are not as expected, use -d !

One example usage could look like this:
pqdl.exe -u leoluk -p my_pass
This will download all available PQs and rename them to name_date.zip.

pqdl.exe -u "leo luk" -p my_pass
This will download all available PQs and rename them to name_date.zip. You can use spaces in the username or password if you put them into quotation marks (many people ask me about this!)

pqdl.exe -u leoluk -p my_pass PQ1 PQ2
This will download the PQs called  "PQ1" und "PQ2" and rename them to name_date.zip.

pqdl.exe -s -u leoluk -p my_pass PQ1 PQ2
This will download the PQs called  "PQ1" und "PQ2" and rename them to name.zip (without date!).

pqdl.exe -s -l -d -u leoluk -p my_pass Freiburg
This will download the PQ called "Freiburg" and output debug data.

pqdl.exe -s -l -d -u leoluk -p my_pass Freiburg
This will simulate to download the PQ called "Freiburg" and output debug data, but it will skip the real download (useful for testing settings).

pqdl.exe -u leoluk -p my_pass #PQ1 #PQ2
This will download all PQs except PQ1 and PQ2.

pqdl.exe -u leoluk -p my_pass -n
This will download all PQs except special PQs like the My Finds-PQ.


pqdl.exe -d -u leoluk -p mein_passwort -n -j -z

This will download all PQs except special PQs like the My Finds-PQ. The option -j tells the program to not download again files that have already been downloaded. The option -z will unzip the files, delete the containers und rename the unzipped files.

Download

The most recent download can be found on the Sourceforge project page.

Please don't upload elsewhere! (they are many updates, and I don't really like old versions floating around everywhere).

Please report every bug you find!

Crashes while parsing the site

Many people report me errors while parsing the site (especially "IndexError"), but I couldn't reproduce any of them, and the problems disappear some time later.
If you encounter such a problem, please immediately make a debug output using -d -t --httpmaindebug and send me the output. Thanks.

Changelog

04.06.2010 - Version 0.1.0

  • initial release

06.06.2010 - Version 0.2.0

  • some small output fixes
  • completely removed colored output
  • fixed problems with special chars in PQ names
  • improved versioning and version output
  • message about spaces in username or password
  • recognition of the My Finds-Pocket Query
  • exclude the My Finds-Pocket Query using -n
  • remove downloaded files from GC.com using -r
  • possibility to download all PQs except some (see examples)
  • --httpremovedebug extended debug

06.06.2010 - Version 0.2.1

  • improved deletion of Pocket Queries by implenting a parser that simulates the client-side javascript code (--ctl switch for debugging)

12.06.2010 - Version 0.2.2

  • adding journal file that prevents the program from downloading a PQ with the same date again; enable it using -j, you can change the database filename with --journalfile, reset it with --resetjournal, for read-only access use --usejournal
  • fixed some help strings and debug output
  • code cleanup
  • fixed problems with old versions of mechanize (thanks, Jan!)
  • adding versions of all used modules to debug output
  • improving internal handling of My Finds-PQ
  • delays between the requests using -e ("hide mode")
  • extended debugging of the parser using --httpmaindebug

12.06.2010 - Version 0.2.3

  • fixed some help strings and debug output
  • fixing bugs related to the journal file
  • unzip the downloaded files with -z (the ZIP files will be removed unless you use --keepzip)
    the files will be recognized automatically and renamed (example: Name-of-PQ_1234567_06-12-2010_waypoints.gpx)

12.06.2010 - Version 0.2.4

  • improved security, you can now just use -u without -p and you will be asked for your password

18.06.2010 - Version 0.3.0

  • sorry, not translated yet (you can look at the cmdline help to see the new features)
  • man kann jetzt zusätzlich zum friendlyname auch noch den Namen (in Anführungszeichen falls Leerzeichen!), das Datum der Erzeugung, die Anzahl der Caches oder die ID des Caches als Argumente angeben bzw. mit # ausschließen. Beispiel: Mit 1000 kann man alle PQs herunterladen, die genau 1000 Caches enthalten.
  • für die Argumente können jetzt Wildcards verwendet werden, nach dem UNIX-Standard (* für irgendein Zeichen, ? für ein Zeichen, [...] für Zeichen in der Klammer oder [!...] um das Zeichen auszuschließen
    durch diese Änderungen können jetzt mehrere Argumente als Filter kombiniert werden, mit W* #*x* kann man z.B. alle PQs herunterladen, die mit W beginnen, aber keine, die x enthalten. 
  • erweiterte Unterstützung für pqloader, mit der Option -m kann aus einer Mappings-Datei jeder PQ einen Prefix zuweisen, also die Datenbank wählen; der Name wird dann dem Dateinamen vorangestellt (Trennzeichen kann mit --sep geändert werden)
  • der nur in manchen Fällen auftretende IndexError-Fehler wurde behoben (das Problem waren übrigens noch nicht erzeugte PQs, deshalb verschwindet das Problem nach einer Weile)
  • mit der Option --myfinds kann eine My Finds-PQ angefordert werden, falls verfügbar (um diese herunterzuladen, ist warscheinlich ein erneuter Aufruf von PqDL  erforderlich, falls die PQ nicht schnell genug erzeugt wird)
  • das Passwort kann in der Kommandozeile weggelassen werden, wenn dies passiert, wird danach gefragt (versteckte Eingabe!)
  • besseres Zurücksetzen der Logdatei
  • bessere Verarbeitung der Logdateien
01.08.2010 - Version 0.3.1
  • Bugfix: GC.com update had broken the parser

Sourcecode (for Linux/Mac/etc.)

plarge

This tool is written in Python and the source code is available here on the Sourceforge project page under the GPL.

I've built an executable for Windows, if you want to run it on other OS, you can use the source code (you should consider doing this on windows too, it's the best choice because you can always update to the latest trunk).

If you want to run this tool from source, you need Python 2.6 and the Python packets mechanize, beautifulsoup, termcolor und colorama. (if you don't know what pip is, try it out, it's a very elegant way to download Python packages ).

This is how to get the latest stable release from SVN:

svn co https://pqdl.svn.sourceforge.net/svnroot/pqdl/tags/pqdl-v0.1 pqdl-v0.1

If you want to use the latest trunk (including the latest features, but the latest bugs too):

svn co https://pqdl.svn.sourceforge.net/svnroot/pqdl/trunk pqdl-trunk

If you want to update the trunk, use svn up in the directory that contains the source code, and it will update it automatically. If you want to use a newer stable release, you need to check it out again. (svn co https://...)

The latest changes in the stable releases are listed here and in the sourceforge release note. If you want to see the latest changes in the trunk, look at the last svn revisions (as example on Trac).

Beta versions

Beta versions are released like stable versions and will appear on the changelog and be compiled for Windows.

Bugs and Feature Requests

If you want to send me bug reports, feature requests, or any other comment, use my contact form or send me a mail (address is written above the contact form).

Please reporty every bug you find and send me the debug output! (use -d)

Comments