B. Appendix: Limits

This section documents some technical limits of CWB. Most of these are due to the design of the index file format (notably the pervasive use of signed 32-bit integer values), but some additional restrictions are imposed by implementation decisions.

• The maximum corpus size is 2,147,483,647 tokens (the largest value that can be stored as a signed 32-bit integer). In the CWB source code, this is represented by the macro CL_MAX_CORPUS_SIZE.
• The maximum size of a p-attribute lexicon is 2,147,483,647 bytes (or 2 GiB), due to the use of signed 32-bit integer offsets into the .lexicon file.¹⁹
• The same 2 GiB limit applies to the “lexicon” storing all distinct annotation strings of an s-attribute with values (the .avs file).
• The maximum length of an annotation string is 4096 bytes, including the NUL terminator. In UTF-8 encoding, a single character may occupy two or more bytes, so the maximum character length of an annotation string may be smaller.

  The length limit is determined by the macro CL_MAX_LINE_LENGTH in the CWB source code and can be increased if absolutely necessary. This is strongly discouraged, as it may create index files that are not compatible with standard builds of CWB.

• The maximum length of a filename is determined by the macro CL_MAX_FILENAME_LENGTH in the CWB source code. Its default value is currently 1024 bytes.
• The maximum length of an input line in the .vrt format is currently 65,536 bytes. It is determined by the macro MAX_INPUT_LINE_LENGTH in the cwb-encode source code.