Advanced Topics - Paperless-ngx (2024)

Paperless offers a couple of features that automate certain tasks and makeyour life easier.

Matching tags, correspondents, document types, and storage paths

Paperless will compare the matching algorithms defined by every tag,correspondent, document type, and storage path in your database to seeif they apply to the text in a document. In other words, if you define atag called Home Utility that had a match property of bc hydro anda matching_algorithm of Exact, Paperless will automatically tagyour newly-consumed document with your Home Utility tag so long as thetext bc hydro appears in the body of the document somewhere.

The matching logic is quite powerful. It supports searching the text ofyour document with different algorithms, and as such, someexperimentation may be necessary to get things right.

In order to have a tag, correspondent, document type, or storage pathassigned automatically to newly consumed documents, assign a match andmatching algorithm using the web interface. These settings define whento assign tags, correspondents, document types, and storage paths todocuments.

The following algorithms are available:

• None: No matching will be performed.
• Any: Looks for any occurrence of any word provided in match inthe PDF. If you define the match as Bank1 Bank2, it will matchdocuments containing either of these terms.
• All: Requires that every word provided appears in the PDF,albeit not in the order provided.
• Exact: Matches only if the match appears exactly as provided(i.e. preserve ordering) in the PDF.
• Regular expression: Parses the match as a regular expression andtries to find a match within the document.
• Fuzzy match: Uses a partial matching based on locating the tag textinside the document, using a partial ratio
• Auto: Tries to automatically match new documents. This does notrequire you to set a match. See the notes below.

When using the any or all matching algorithms, you can search forterms that consist of multiple words by enclosing them in double quotes.For example, defining a match text of "Bank of America" BofA using theany algorithm, will match documents that contain either "Bank ofAmerica" or "BofA", but will not match documents containing "Bank ofSouth America".

Then just save your tag, correspondent, document type, or storage pathand run another document through the consumer. Once complete, you shouldsee the newly-created document, automatically tagged with theappropriate data.

Automatic matching

Paperless-ngx comes with a new matching algorithm called Auto. Thismatching algorithm tries to assign tags, correspondents, document types,and storage paths to your documents based on how you have alreadyassigned these on existing documents. It uses a neural network under thehood.

If, for example, all your bank statements of your account 123 at theBank of America are tagged with the tag "bofa123" and the matchingalgorithm of this tag is set to Auto, this neural network will examineyour documents and automatically learn when to assign this tag.

Paperless tries to hide much of the involved complexity with thisapproach. However, there are a couple caveats you need to keep in mindwhen using this feature:

• Changes to your documents are not immediately reflected by thematching algorithm. The neural network needs to be trained on yourdocuments after changes. Paperless periodically (default: once eachhour) checks for changes and does this automatically for you.
• The Auto matching algorithm only takes documents into account whichare NOT placed in your inbox (i.e. have any inbox tags assigned tothem). This ensures that the neural network only learns fromdocuments which you have correctly tagged before.
• The matching algorithm can only work if there is a correlationbetween the tag, correspondent, document type, or storage path andthe document itself. Your bank statements usually contain your bankaccount number and the name of the bank, so this works reasonablywell, However, tags such as "TODO" cannot be automaticallyassigned.
• The matching algorithm needs a reasonable number of documents toidentify when to assign tags, correspondents, storage paths, andtypes. If one out of a thousand documents has the correspondent"Very obscure web shop I bought something five years ago", it willprobably not assign this correspondent automatically if you buysomething from them again. The more documents, the better.
• Paperless also needs a reasonable amount of negative examples todecide when not to assign a certain tag, correspondent, documenttype, or storage path. This will usually be the case as you startfilling up paperless with documents. Example: If all your documentsare either from "Webshop" or "Bank", paperless will assign oneof these correspondents to ANY new document, if both are set toautomatic matching.

Hooking into the consumption process

Sometimes you may want to do something arbitrary whenever a document isconsumed. Rather than try to predict what you may want to do, Paperlesslets you execute scripts of your own choosing just before or after adocument is consumed using a couple of simple hooks.

Just write a script, put it somewhere that Paperless can read & execute,and then put the path to that script in paperless.conf ordocker-compose.env with the variable name of eitherPAPERLESS_PRE_CONSUME_SCRIPT or PAPERLESS_POST_CONSUME_SCRIPT.

Pre-consumption script

Executed after the consumer sees a new document in the consumptionfolder, but before any processing of the document is performed. Thisscript can access the following relevant environment variables set:

A simple but common example for this would be creating a simple scriptlike this:

/usr/local/bin/ocr-pdf

#!/usr/bin/env bashpdf2pdfocr.py -i ${DOCUMENT_WORKING_PATH}

/etc/paperless.conf

...PAPERLESS_PRE_CONSUME_SCRIPT="/usr/local/bin/ocr-pdf"...

This will pass the path to the document about to be consumed to/usr/local/bin/ocr-pdf, which will in turn callpdf2pdfocr.py on yourdocument, which will then overwrite the file with an OCR'd version ofthe file and exit. At which point, the consumption process will beginwith the newly modified file.

The script's stdout and stderr will be logged line by line to thewebserver log, along with the exit code of the script.

Post-consumption script

Executed after the consumer has successfully processed a document andhas moved it into paperless. It receives the following environmentvariables:

The script can be in any language, A simple shell script example:

#!/usr/bin/env bashecho "A document with an id of ${DOCUMENT_ID} was just consumed. I know thefollowing additional information about it:* Generated File Name: ${DOCUMENT_FILE_NAME}* Archive Path: ${DOCUMENT_ARCHIVE_PATH}* Source Path: ${DOCUMENT_SOURCE_PATH}* Created: ${DOCUMENT_CREATED}* Added: ${DOCUMENT_ADDED}* Modified: ${DOCUMENT_MODIFIED}* Thumbnail Path: ${DOCUMENT_THUMBNAIL_PATH}* Download URL: ${DOCUMENT_DOWNLOAD_URL}* Thumbnail URL: ${DOCUMENT_THUMBNAIL_URL}* Correspondent: ${DOCUMENT_CORRESPONDENT}* Tags: ${DOCUMENT_TAGS}It was consumed with the passphrase ${PASSPHRASE}"

The script's stdout and stderr will be logged line by line to thewebserver log, along with the exit code of the script.

Docker

To hook into the consumption process when using Docker, youwill need to pass the scripts into the container via a host mountin your docker-compose.yml.

Assuming you have/home/paperless-ngx/scripts/post-consumption-example.sh as ascript which you'd like to run.

You can pass that script into the consumer container via a host mount:

...webserver: ... volumes: ... - /home/paperless-ngx/scripts:/path/in/container/scripts/ # (1)! environment: # (3)! ... PAPERLESS_POST_CONSUME_SCRIPT: /path/in/container/scripts/post-consumption-example.sh # (2)!...

1. The external scripts directory is mounted to a location inside the container.
2. The internal location of the script is used to set the script to run
3. This can also be set in docker-compose.env

Troubleshooting:

• Monitor the Docker Compose logcd ~/paperless-ngx; docker compose logs -f
• Check your script's permission e.g. in case of permission errorsudo chmod 755 post-consumption-example.sh
• Pipe your scripts's output to a log file e.g.echo "${DOCUMENT_ID}" | tee --append /usr/src/paperless/scripts/post-consumption-example.log

File name handling

By default, paperless stores your documents in the media directory andrenames them using the identifier which it has assigned to eachdocument. You will end up getting files like 0000123.pdf in your mediadirectory. This isn't necessarily a bad thing, because you normallydon't have to access these files manually. However, if you wish to nameyour files differently, you can do that by adjusting thePAPERLESS_FILENAME_FORMAT configuration optionor using storage paths (see below). Paperless adds thecorrect file extension e.g. .pdf, .jpg automatically.

This variable allows you to configure the filename (folders are allowed)using placeholders. For example, configuring this to

PAPERLESS_FILENAME_FORMAT={created_year}/{correspondent}/{title}

will create a directory structure as follows:

2019/ My bank/ Statement January.pdf Statement February.pdf2020/ My bank/ Statement January.pdf Letter.pdf Letter_01.pdf Shoe store/ My new shoes.pdf

Placeholders

Paperless provides the following placeholders within filenames:

• {asn}: The archive serial number of the document, or "none".
• {correspondent}: The name of the correspondent, or "none".
• {document_type}: The name of the document type, or "none".
• {tag_list}: A comma separated list of all tags assigned to thedocument.
• {title}: The title of the document.
• {created}: The full date (ISO format) the document was created.
• {created_year}: Year created only, formatted as the year withcentury.
• {created_year_short}: Year created only, formatted as the yearwithout century, zero padded.
• {created_month}: Month created only (number 01-12).
• {created_month_name}: Month created name, as per locale
• {created_month_name_short}: Month created abbreviated name, as perlocale
• {created_day}: Day created only (number 01-31).
• {added}: The full date (ISO format) the document was added topaperless.
• {added_year}: Year added only.
• {added_year_short}: Year added only, formatted as the year withoutcentury, zero padded.
• {added_month}: Month added only (number 01-12).
• {added_month_name}: Month added name, as per locale
• {added_month_name_short}: Month added abbreviated name, as perlocale
• {added_day}: Day added only (number 01-31).
• {owner_username}: Username of document owner, if any, or "none"
• {original_name}: Document original filename, minus the extension, if any, or "none"
• {doc_pk}: The paperless identifier (primary key) for the document.

Paperless will try to conserve the information from your database asmuch as possible. However, some characters that you can use in documenttitles and correspondent names (such as : \ / and a couple more) arenot allowed in filenames and will be replaced with dashes.

If paperless detects that two documents share the same filename,paperless will automatically append _01, _02, etc to the filename.This happens if all the placeholders in a filename evaluate to the samevalue.

If there are any errors in the placeholders included in PAPERLESS_FILENAME_FORMAT,paperless will fall back to using the default naming scheme instead.

PAPERLESS_FILENAME_FORMAT=../../my/custom/location/{title}

Empty placeholders

You can affect how empty placeholders are treated by changing thePAPERLESS_FILENAME_FORMAT_REMOVE_NONE setting.

Enabling this results in all empty placeholders resolving to "" instead of "none" as stated above. Spacesbefore empty placeholders are removed as well, empty directories are omitted.

Storage paths

When a single storage layout is not sufficient for your use case, storage paths allow for more complexstructure to set precisely where each document is stored in the file system.

• Each storage path is a PAPERLESS_FILENAME_FORMAT andfollows the rules described above
• Each document is assigned a storage path using the matching algorithms described above, but can beoverwritten at any time

For example, you could define the following two storage paths:

1. Normal communications are put into a folder structure sorted byyear/correspondent
2. Communications with insurance companies are stored in a flatstructure with longer file names, but containing the full date ofthe correspondence.

By Year = {created_year}/{correspondent}/{title}Insurances = Insurances/{correspondent}/{created_year}-{created_month}-{created_day} {title}

If you then map these storage paths to the documents, you might get thefollowing result. For simplicity, By Year defines the samestructure as in the previous example above.

2019/ # By Year My bank/ Statement January.pdf Statement February.pdfInsurances/ # Insurances Healthcare 123/ 2022-01-01 Statement January.pdf 2022-02-02 Letter.pdf 2022-02-03 Letter.pdf Dental 456/ 2021-12-01 New Conditions.pdf

Celery Monitoring

The monitoring toolFlower can be usedto view more detailed information about the health of the celery workersused for asynchronous tasks. This includes details on currently running,queued and completed tasks, timing and more. Flower can also be usedwith Prometheus, as it exports metrics. For details on its capabilities,refer to the Flowerdocumentation.

Flower can be enabled with the setting PAPERLESS_ENABLE_FLOWER.To configure Flower further, create a flowerconfig.py andplace it into the src/paperless directory. For a Dockerinstallation, you can use volumes to accomplish this:

services: # ... webserver: environment: - PAPERLESS_ENABLE_FLOWER ports: - 5555:5555 # (2)! # ... volumes: - /path/to/my/flowerconfig.py:/usr/src/paperless/src/paperless/flowerconfig.py:ro # (1)!

1. Note the :ro tag means the file will be mounted as read only.
2. By default, Flower runs on port 5555, but this can be configured.

Custom Container Initialization

The Docker image includes the ability to run custom user scripts duringstartup. This could be utilized for installing additional tools orPython packages, for example. Scripts are expected to be shell scripts.

To utilize this, mount a folder containing your scripts to the custominitialization directory, /custom-cont-init.d and placescripts you wish to run inside. For security, the folder must be ownedby root and should have permissions of a=rx. Additionally, scriptsmust only be writable by root.

Your scripts will be run directly before the webserver completesstartup. Scripts will be run by the root user.If you would like to switch users, the utility gosu is available andpreferred over sudo.

This is an advanced functionality with which you could break functionalityor lose data. If you experience issues, please disable any custom scriptsand try again before reporting an issue.

For example, using Docker Compose:

services: # ... webserver: # ... volumes: - /path/to/my/scripts:/custom-cont-init.d:ro # (1)!

1. Note the :ro tag means the folder will be mounted as read only. This is for extra security against changes

MySQL Caveats

Case Sensitivity

The database interface does not provide a method to configure a MySQLdatabase to be case sensitive. This would prevent a user from creating atag Name and NAME as they are considered the same.

Per Django documentation, to enable this requires manual intervention.To enable case sensitive tables, you can execute the following commandagainst each table:

ALTER TABLE <table_name> CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;

You can also set the default for new tables (this does NOT affectexisting tables) with:

ALTER DATABASE <db_name> CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;

Missing timezones

MySQL as well as MariaDB do not have any timezone information by default (though somedocker images such as the official MariaDB image take care of this for you) which willcause unexpected behavior with date-based queries.

To fix this, execute one of the following commands:

MySQL: mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root mysql -p

MariaDB: mariadb-tzinfo-to-sql /usr/share/zoneinfo | mariadb -u root mysql -p

Barcodes

Paperless is able to utilize barcodes for automatically performing some tasks.

At this time, the library utilized for detection of barcodes supports the following types:

• AN-13/UPC-A
• UPC-E
• EAN-8
• Code 128
• Code 93
• Code 39
• Codabar
• Interleaved 2 of 5
• QR Code
• SQ Code

You may check for updates on the zbar library homepage.For usage in Paperless, the type of barcode does not matter, only the contents of it.

For how to enable barcode usage, see the configuration.The two settings may be enabled independently, but do have interactions as explainedbelow.

Document Splitting

When enabled, Paperless will look for a barcode with the configured value and create a new documentstarting from the next page. The page with the barcode on it will not be retained. Itis expected to be a page existing only for triggering the split.

Archive Serial Number Assignment

When enabled, the value of the barcode (as an integer) will be used to set the document'sarchive serial number, allowing quick reference back to the original, paper document.

If document splitting via barcode is also enabled, documents will be split when an ASNbarcode is located. However, differing from the splitting, the page with thebarcode will be retained. This allows application of a barcode to any page, includingone which holds data to keep in the document.

Tag Assignment

When enabled, Paperless will parse barcodes and attempt to interpret and assign tags.

See the relevant settings PAPERLESS_CONSUMER_ENABLE_TAG_BARCODEand PAPERLESS_CONSUMER_TAG_BARCODE_MAPPINGfor more information.

Automatic collation of double-sided documents

This feature is turned off by default, see configuration on how to turn it on.

Summary

If you have a scanner with an automatic document feeder (ADF) that only scans a single side,this feature makes scanning double-sided documents much more convenient by automaticallycollating two separate scans into one document, reordering the pages as necessary.

Usage example

Suppose you have a double-sided document with 6 pages (3 sheets of paper). First,put the stack into your ADF as normal, ensuring that page 1 is scanned first. Your ADFwill now scan pages 1, 3, and 5. Then you (or your scanner, if it supports it) uploadthe scan into the correct sub-directory of the consume folder (double-sided by default;keep in mind that Paperless will not automatically create the directory for you.)Paperless will then process the scan and move it into an internal staging area.

The next step is to turn your stack upside down (without reordering the sheets of paper),and scan it once again, your ADF will now scan pages 6, 4, and 2, in that order. Once thisscan is copied into the sub-directory, Paperless will collate the previous scan with thenew one, reversing the order of the pages on the second, "even numbered" scan. Theresulting document will have the pages 1-6 in the correct order, and this new file willthen be processed as normal.

Things that could go wrong

Paperless will notice when the first, "odd numbered" scan has less pages than the secondscan (this can happen when e.g. the ADF skipped a few pages in the first pass). In thatcase, Paperless will remove the staging copy as well as the scan, and give you an errormessage asking you to restart the process from scratch, by scanning the odd pages again,followed by the even pages.

It's important that the scan files get consumed in the correct order, and one at a time.You therefore need to make sure that Paperless is running while you upload the files intothe directory; and if you're using polling, make sure thatCONSUMER_POLLING is set to a value lower than it takes for the second scan to appear,like 5-10 or even lower.

Another thing that might happen is that you start a double sided scan, but then forgetto upload the second file. To avoid collating the wrong documents if you then come backa day later to scan a new double-sided document, Paperless will only keep an "odd numberedpages" file for up to 30 minutes. If more time passes, it will consider the next incomingscan a completely new "odd numbered pages" one. The old staging file will get discarded.

Interaction with "subdirs as tags"

The collation feature can be used together with the subdirs as tagsfeature (but this is not a requirement). Just create a correctly named double-sided subdirin the hierarchy and upload your scans there. For example, both double-sided/foo/bar aswell as foo/bar/double-sided will cause the collated document to be treated as if itwere uploaded into foo/bar and receive both foo and bar tags, but not double-sided.

Interaction with document splitting

You can use the document splitting feature, but if you use a normalsingle-sided split marker page, the split document(s) will have an empty page at the front (orwhatever else was on the backside of the split marker page.) You can work around that by havinga split marker page that has the split barcode on both sides. This way, the extra page willget automatically removed.

SSO and third party authentication with Paperless-ngx

Paperless-ngx has a built-in authentication system from Django but you can easily integrate anexternal authentication solution using one of the following methods:

Remote User authentication

This is a simple option that uses remote user authentication made available by certain SSOapplications. See the relevant configuration options for more information:PAPERLESS_ENABLE_HTTP_REMOTE_USER,PAPERLESS_HTTP_REMOTE_USER_HEADER_NAMEand PAPERLESS_LOGOUT_REDIRECT_URL

OpenID Connect and social authentication

Version 2.5.0 of Paperless-ngx added support for integrating other authentication systems viathe django-allauth package. Once set up, userscan either log in or (optionally) sign up using any third party systems you integrate. See therelevant configuration settings anddjango-allauth docsfor more information.

To associate an existing Paperless-ngx account with a social account, first login with yourregular credentials and then choose "My Profile" from the user dropdown in the app and youwill see options to connect social account(s). If enabled, signup options will be availableon the login page.

As an example, to set up login via Github, the following environment variables would need to beset:

PAPERLESS_APPS="allauth.socialaccount.providers.github"PAPERLESS_SOCIALACCOUNT_PROVIDERS='{"github": {"APPS": [{"provider_id": "github","name": "Github","client_id": "<CLIENT_ID>","secret": "<CLIENT_SECRET>"}]}}'

Or, to use OpenID Connect ("OIDC"), via Keycloak in this example:

PAPERLESS_APPS="allauth.socialaccount.providers.openid_connect"PAPERLESS_SOCIALACCOUNT_PROVIDERS='{"openid_connect": {"APPS": [{"provider_id": "keycloak","name": "Keycloak","client_id": "paperless","secret": "<CLIENT_SECRET>","settings": { "server_url": "https://<KEYCLOAK_SERVER>/realms/<REALM>/.well-known/openid-configuration"}}]}}'

More details about configuration option for various providers can be found in the allauth documentation.

Disabling Regular Login

Once external auth is set up, 'regular' login can be disabled with the PAPERLESS_DISABLE_REGULAR_LOGIN setting.