7.1 Running CQP as a backend

• CQP is a useful tool for interactive work, but many tasks become tedious when they have to be carried out by hand; macros can be used as templates, providing some relief; however, full scripting is still desirable (and in some cases essential)

• similarly, the output of CQP requires post-processing at times: better formatting of KWIC lines (especially for HTML output), different sort options for frequency tables, frequency counts on normalised word forms (or other transformations of the values)

• for both purposes, an external scripting tool or programming language is required, which has to interact dynamically with CQP (which acts as a query engine)

• CQP provides some support for such interfaces: when invoked with the -c flag, it switches to child mode (which could also be called “slave” mode):
  • the init file .cqprc (in the current user's home folder) is not automatically read at startup
  • CQP prints its version number after startup
  • all interactive features are deactivated (paged display and highlighting)
  • query results are not automatically displayed (set AutoShow off;)
  • after the execution of a command, CQP flushes output buffers (so that the interface will not hang waiting for output from the command)
  • in case of a syntax error, the string PARSE ERROR is printed on stderr
  • the special command .EOL.; can be used to insert the line

    -::-EOL-::-

    as a marker into CQP's output
  • when the ProgressBar option is activated, progress messages are not echoed in a single screen line (using carriage returns) on stderr, but rather printed in separate lines on stdout; these lines have the standardized format

    -::-PROGRESS-::- TAB pass TAB no. of passes TAB progress message

• the CWB/Perl interface makes use of all these features to provide an efficient and robust interface between a Perl script and the CQP backend

• these features are also used extensively by CQPweb and many other web interfaces

• the output of many CQP commands is neatly formatted for human readers; this pretty printing feature can be switched off with the command

  > set PrettyPrint off;

  the output of the show, group and count commands now has a simple and standardized format that can more easily be parsed by the invoking program; output formats for the different uses of the show command are documented below; see Section 7.3 for the output formats of group and count

  • show corpora; prints the names of all available corpora on separate lines, in alphabetical order
  • show named; lists all named query results on separate lines in the format

    flags TAB query name TAB no. of matches

    flags is a three-character code representing the flags m = stored in memory, d = saved to disk, * = modified since last saved; flags that are not set are shown as - characters; query name is the long name of the query result, i.e. it has the form corpus:name; when a query result has not yet been loaded from disk, the no. of matches cannot be determined and is reported as 0
  • show; concatenates the output of show corpora; and show named; without any separator; it is recommended to invoke the two commands separately when using CQP as a backend
  • show active; prints the name of the currently active corpus on a line on its own (this is in fact available when using CQP interactively, albeit useless because the active corpus is displayed in the CQP command prompt!)
  • show cd; lists all attributes that are defined for the currently active corpus; each attribute is printed on a separate line with the format

    attribute type TAB attribute name $[$ TAB $[$ -V $]$ $]$

    attribute type is one of the strings p-Att (positional attribute), s-Att (structural attribute) or a-Att (alignment attribute), so the attribute type can easily be recognized from the first character of the output line; the third column is only printed for s-attributes and is either an empty string (no annotations) or -V (regions have annotated values)
  • new in CQP v3.4.18: the output of show cd; now always prints four TAB-separated columns, i.e. lines of the form

    attribute type TAB attribute name TAB $[$ -V $]$ TAB $[$ * $]$

    the third column is -V for an s-attribute with annotations and an empty string otherwise; the fourth column is * if the attribute is currently selected for display and an empty string otherwise

• the CWB/Perl interface automatically deactivates pretty printing

• running CQP as a backend can be a security risk, e.g. when queries submitted to a Web server are passed through to the CQP process unaltered; this may allow malicious users to execute arbitrary shell commands on the Web server

• as a safeguard against such attacks, CQP provides a query lock mode, which allows only queries to be executed, while all other commands (including cat, sort, group, etc.) are blocked

• the query lock mode is activated with the command

  > set QueryLock $n$;

  where $n$ is a randomly chosen integer number; it can only be de-activated with

  > unlock $n$;

  using the same number $n$

• new in CQP v3.4.18: A frontend that intends to parse KWIC output from the cat command should normally change the default slash (/) separator between p-attributes to something less ambiguous. For example, a KWIC output line with word and lemma activated might look as follows:

  $\implies$ <s> Most/most CP/M/CP/M sytems/system ...

  The new CQP option AttributeSeparator or (sep) allows users to override this default setting with an (almost) arbitrary string, e.g.

  > set AttributeSeparator "__";
  $\implies$ <s> Most__most CP/M__CP/M sytems__system ...

• new in CQP v3.4.24: Likewise, a frontend might well prefer to use a different separator between tokens in the KWIC in place of the default space (since space is a legal character within p-attribute values); the new option TokenSeparator or tok parallels AttributeSeparator to override the default space with (almost) anything.

• The safest option for unambiguous attribute or token separator strings is to use a control code that is disallowed in attribute values, e.g. TAB (#9), BEL (#7) or ESC (#27).¹⁷ Note that these controls have to be included as literal characters in the set command because CQP doesn't support escape sequences such as \t or \x09.

• An overriding AttributeSeparator or TokenSeparator can be turned off by setting the option in question to an empty string. This re-enables the default (slash or space respectively).