Paperless offers a couple features that automate certain tasks and make your lifeeasier.
Matching tags, correspondents, document types, and storage paths
Paperless will compare the matching algorithms defined by every tag, correspondent,document type, and storage path in your database to see if they apply to the textin a document. In other words, if you define a tag called Home Utilitythat had a match property of bc hydro and a matching_algorithm ofliteral, Paperless will automatically tag your newly-consumed document withyour Home Utility tag so long as the text bc hydro appears in the bodyof the document somewhere.
The matching logic is quite powerful. It supports searching the text of yourdocument with different algorithms, and as such, some experimentation may benecessary to get things right.
In order to have a tag, correspondent, document type, or storage path assignedautomatically to newly consumed documents, assign a match and matching algorithmusing the web interface. These settings define when to assign tags, correspondents,document types, and storage paths to documents.
The following algorithms are available:
Any: Looks for any occurrence of any word provided in match in the PDF.If you define the match as
Bank1 Bank2, it will match documents containingeither of these terms.All: Requires that every word provided appears in the PDF, albeit not in theorder provided.
Literal: Matches only if the match appears exactly as provided (i.e. preserve ordering) in the PDF.
Regular expression: Parses the match as a regular expression and tries tofind a match within the document.
Fuzzy match: I don’t know. Look at the source.
Auto: Tries to automatically match new documents. This does not require youto set a match. See the notes below.
When using the any or all matching algorithms, you can search for termsthat consist of multiple words by enclosing them in double quotes. For example,defining a match text of "Bank of America" BofA using the any algorithm,will match documents that contain either “Bank of America” or “BofA”, but willnot match documents containing “Bank of South America”.
Then just save your tag, correspondent, document type, or storage path and runanother document through the consumer. Once complete, you should see thenewly-created document, automatically tagged with the appropriate data.
Automatic matching
Paperless-ngx comes with a new matching algorithm called Auto. This matchingalgorithm tries to assign tags, correspondents, document types, and storage pathsto your documents based on how you have already assigned these on existing documents.It uses a neural network under the hood.
If, for example, all your bank statements of your account 123 at the Bank ofAmerica are tagged with the tag “bofa_123” and the matching algorithm of thistag is set to Auto, this neural network will examine your documents andautomatically learn when to assign this tag.
Paperless tries to hide much of the involved complexity with this approach.However, there are a couple caveats you need to keep in mind when using thisfeature:
Changes to your documents are not immediately reflected by the matchingalgorithm. The neural network needs to be trained on your documents afterchanges. Paperless periodically (default: once each hour) checks for changesand does this automatically for you.
The Auto matching algorithm only takes documents into account which are NOTplaced in your inbox (i.e. have any inbox tags assigned to them). This ensuresthat the neural network only learns from documents which you have correctlytagged before.
The matching algorithm can only work if there is a correlation between thetag, correspondent, document type, or storage path and the document itself.Your bank statements usually contain your bank account number and the nameof the bank, so this works reasonably well, However, tags such as “TODO”cannot be automatically assigned.
The matching algorithm needs a reasonable number of documents to identify whento assign tags, correspondents, storage paths, and types. If one out of athousand documents has the correspondent “Very obscure web shop I boughtsomething five years ago”, it will probably not assign this correspondentautomatically if you buy something from them again. The more documents, the better.
Paperless also needs a reasonable amount of negative examples to decide whennot to assign a certain tag, correspondent, document type, or storage path. This willusually be the case as you start filling up paperless with documents.Example: If all your documents are either from “Webshop” and “Bank”, paperlesswill assign one of these correspondents to ANY new document, if both are setto automatic matching.
Hooking into the consumption process
Sometimes you may want to do something arbitrary whenever a document isconsumed. Rather than try to predict what you may want to do, Paperless letsyou execute scripts of your own choosing just before or after a document isconsumed using a couple simple hooks.
Just write a script, put it somewhere that Paperless can read & execute, andthen put the path to that script in paperless.conf or docker-compose.env with the variable nameof either PAPERLESS_PRE_CONSUME_SCRIPT orPAPERLESS_POST_CONSUME_SCRIPT.
Important
These scripts are executed in a blocking process, which means that ifa script takes a long time to run, it can significantly slow down yourdocument consumption flow. If you want things to run asynchronously,you’ll have to fork the process in your script and exit.
Pre-consumption script
Executed after the consumer sees a new document in the consumption folder, butbefore any processing of the document is performed. This script can access thefollowing relevant environment variables set:
DOCUMENT_SOURCE_PATH
A simple but common example for this would be creating a simple script likethis:
/usr/local/bin/ocr-pdf
#!/usr/bin/env bashpdf2pdfocr.py -i ${DOCUMENT_SOURCE_PATH}
/etc/paperless.conf
...PAPERLESS_PRE_CONSUME_SCRIPT="/usr/local/bin/ocr-pdf"...
This will pass the path to the document about to be consumed to /usr/local/bin/ocr-pdf,which will in turn call pdf2pdfocr.py on your document, which will thenoverwrite the file with an OCR’d version of the file and exit. At which point,the consumption process will begin with the newly modified file.
Post-consumption script
Executed after the consumer has successfully processed a document and has moved itinto paperless. It receives the following environment variables:
DOCUMENT_IDDOCUMENT_FILE_NAMEDOCUMENT_CREATEDDOCUMENT_MODIFIEDDOCUMENT_ADDEDDOCUMENT_SOURCE_PATHDOCUMENT_ARCHIVE_PATHDOCUMENT_THUMBNAIL_PATHDOCUMENT_DOWNLOAD_URLDOCUMENT_THUMBNAIL_URLDOCUMENT_CORRESPONDENTDOCUMENT_TAGS
The script can be in any language, but for a simple shell scriptexample, you can take a look at post-consumption-example.sh in this project.
The post consumption script cannot cancel the consumption process.
Docker
Assumed you have /home/foo/paperless-ngx/scripts/post-consumption-example.sh.
You can pass that script into the consumer container via a host mount in your docker-compose.yml.
...consumer: ... volumes: ... - /home/paperless-ngx/scripts:/path/in/container/scripts/...
Example (docker-compose.yml): - /home/foo/paperless-ngx/scripts:/usr/src/paperless/scripts
which in turn requires the variable PAPERLESS_POST_CONSUME_SCRIPT in docker-compose.env to point to /path/in/container/scripts/post-consumption-example.sh.
Example (docker-compose.env): PAPERLESS_POST_CONSUME_SCRIPT=/usr/src/paperless/scripts/post-consumption-example.sh
Troubleshooting:
Monitor the docker-compose log
cd ~/paperless-ngx; docker-compose logs -fCheck your script’s permission e.g. in case of permission error
sudo chmod 755 post-consumption-example.shSee AlsoBasic Usage - Paperless-ngxPipe your scripts’s output to a log file e.g.
echo "${DOCUMENT_ID}" | tee --append /usr/src/paperless/scripts/post-consumption-example.log
File name handling
By default, paperless stores your documents in the media directory and renames themusing the identifier which it has assigned to each document. You will end up gettingfiles like 0000123.pdf in your media directory. This isn’t necessarily a badthing, because you normally don’t have to access these files manually. However, ifyou wish to name your files differently, you can do that by adjusting thePAPERLESS_FILENAME_FORMAT configuration option.
This variable allows you to configure the filename (folders are allowed) usingplaceholders. For example, configuring this to
PAPERLESS_FILENAME_FORMAT={created_year}/{correspondent}/{title}
will create a directory structure as follows:
2019/ My bank/ Statement January.pdf Statement February.pdf2020/ My bank/ Statement January.pdf Letter.pdf Letter_01.pdf Shoe store/ My new shoes.pdf
Danger
Do not manually move your files in the media folder. Paperless remembers thelast filename a document was stored as. If you do rename a file, paperless willreport your files as missing and won’t be able to find them.
Paperless provides the following placeholders within filenames:
{asn}: The archive serial number of the document, or “none”.{correspondent}: The name of the correspondent, or “none”.{document_type}: The name of the document type, or “none”.{tag_list}: A comma separated list of all tags assigned to the document.{title}: The title of the document.{created}: The full date (ISO format) the document was created.{created_year}: Year created only.{created_month}: Month created only (number 01-12).{created_day}: Day created only (number 01-31).{added}: The full date (ISO format) the document was added to paperless.{added_year}: Year added only.{added_month}: Month added only (number 01-12).{added_day}: Day added only (number 01-31).
Paperless will try to conserve the information from your database as much as possible.However, some characters that you can use in document titles and correspondent names (suchas : \ / and a couple more) are not allowed in filenames and will be replaced with dashes.
If paperless detects that two documents share the same filename, paperless will automaticallyappend _01, _02, etc to the filename. This happens if all the placeholders in a filenameevaluate to the same value.
Hint
You can affect how empty placeholders are treated by changing the following setting totrue.
PAPERLESS_FILENAME_FORMAT_REMOVE_NONE=True
Doing this results in all empty placeholders resolving to “” instead of “none” as stated above.Spaces before empty placeholders are removed as well, empty directories are omitted.
Hint
Paperless checks the filename of a document whenever it is saved. Therefore,you need to update the filenames of your documents and move them after alteringthis setting by invoking the document renamer.
Warning
Make absolutely sure you get the spelling of the placeholders right, or elsepaperless will use the default naming scheme instead.
Caution
As of now, you could totally tell paperless to store your files anywhere outsidethe media directory by setting
PAPERLESS_FILENAME_FORMAT=../../my/custom/location/{title}
However, keep in mind that inside docker, if files get stored outside of thepredefined volumes, they will be lost after a restart of paperless.
Storage paths
One of the best things in Paperless is that you can not only access the documents via theweb interface, but also via the file system.
When as single storage layout is not sufficient for your use case, storage paths come tothe rescue. Storage paths allow you to configure more precisely where each document is storedin the file system.
Each storage path is a PAPERLESS_FILENAME_FORMAT and follows the rules described above
Each document is assigned a storage path using the matching algorithms described above, butcan be overwritten at any time
For example, you could define the following two storage paths:
Normal communications are put into a folder structure sorted by year/correspondent
Communications with insurance companies are stored in a flat structure with longer file names,but containing the full date of the correspondence.
By Year = {created_year}/{correspondent}/{title}Insurances = Insurances/{correspondent}/{created_year}-{created_month}-{created_day} {title}
If you then map these storage paths to the documents, you might get the following result.For simplicity, By Year defines the same structure as in the previous example above.
2019/ # By Year My bank/ Statement January.pdf Statement February.pdf Insurances/ # Insurances Healthcare 123/ 2022-01-01 Statement January.pdf 2022-02-02 Letter.pdf 2022-02-03 Letter.pdf Dental 456/ 2021-12-01 New Conditions.pdf
Hint
Defining a storage path is optional. If no storage path is defined for a document, the globalPAPERLESS_FILENAME_FORMAT is applied.
Caution
If you adjust the format of an existing storage path, old documents don’t get relocated automatically.You need to run the document renamer to adjust their pathes.
