← All projects

Research tooling · Built by Sayan Mondal

PaperHarbor

Turn scattered references into an organized collection of available open-access PDFs.

CategoryResearch productivity
InterfaceLocal web UI & CLI
OutputPDF archive & CSV

For individual papers and bibliography batches.

Project overview

Less time collecting papers.

The problem

Finding full texts, naming files, and tracking missing papers interrupts literature review. A bibliography may contain a mixture of publisher links, repository copies, and references with incomplete metadata, making manual collection repetitive.

The approach

Import Zotero, BibTeX, CSL-JSON, or a DOI/URL. Search and select references locally, then download available copies. The same discovery pipeline supports interactive selection and repeatable command-line batches.

Supported inputs

Start with one paper or an existing library.

Choose the source that fits your research workflow
SourceInputTypical use
Single paperDOI or article URL; optional fallback titleRetrieve a specific reference without a library.
Local ZoteroAuto-detected or explicit SQLite database pathWork through missing attachments in a collection.
Bibliography exportBibTeX or CSL-JSON fileBuild an archive from an exported reading list.
Zotero Web APIAPI key and personal or group library IDRead references from an online library.
Previous reportPaperHarbor CSVRevisit unresolved references after an earlier run.

In the browser interface, search references, filter collections, select papers, and follow progress and logs before reviewing the final report.

Architecture

From references to a personal archive.

  1. Read metadata
  2. Select references
  3. Discover candidates
  4. Validate downloads
  5. Organize PDFs
  6. Report outcomes
Discovery and fallbacks

The provider aggregator checks reviewed author copies and supported publisher links, then conditionally queries Unpaywall, OpenAlex, Europe PMC, arXiv, Semantic Scholar, and Crossref. If initial downloads fail, it queries providers skipped earlier, including search fallbacks, and inspects publisher pages.

Candidate URLs are deduplicated, direct PDFs receive priority over landing pages, and conflicting DOI candidates are rejected. Provider errors are recorded while other searches continue.

Shared orchestration

The application coordinates readers, provider searches, landing-page resolution, file naming, downloads, and reporting. It checks existing attachments and destination files before searching. Repeated DOI outcomes within a batch are reused to avoid duplicate network requests.

References are processed sequentially with a default 0.5-second pause. An individual reference error is recorded and processing continues with the next paper.

Implementation: application coordinator ↗ and provider aggregator ↗.

Download reliability

Check the response before keeping the file.

File and identity checks
  • Require a PDF signature within the first 1,024 bytes and a minimum download size of 1,024 bytes.
  • Reject redirects that identify a conflicting DOI.
  • For candidates requiring an identity check, verify the requested title against text from the PDF’s first page.
  • Stream to a temporary file before moving a successful download into its destination.
Bounded retries

Transient network failures and selected HTTP responses receive up to two retries by default. Backoff uses the server’s Retry-After value when available; requested waits above 30 seconds are deferred for a later attempt.

Access-denied responses are not repeatedly retried. A PDF header check identifies the file format; it does not establish the scientific quality or completeness of a paper.

Implementation: PDF downloader ↗.

Reporting and recovery

Keep the outcome of every reference visible.

A timestamped CSV records bibliographic metadata, status, source provider, file path, size, PDF URL, and notes. Attempt counts and structured attempt logs preserve the discovery history; recovery actions and suggested filenames support manual follow-up.

Outcome categories
  • Downloaded or already present in Zotero or the destination.
  • No open copy found, download failed, or manual access required.
  • Reference error, recognized web resource, or withdrawn paper.
Retry only unresolved work

Pass a previous report with --retry-report to revisit failed, manual-access, and error rows. Add --retry-not-found to include papers for which no copy was found. Verified alternate URLs can be supplied through --pdf-overrides.

Implementation: CSV reporter ↗.

Technology stack

A Python tool with a local interface.

PythonRequestsBeautiful SouppypdfSQLiteZotero Web API

Requests handles HTTP access, Beautiful Soup supports HTML parsing, and pypdf supports PDF inspection. The web server uses Python’s standard library.

Try the project

Use the browser or run a repeatable batch.

Clone the repository and install dependencies in a virtual environment. The following commands use Windows PowerShell:

git clone https://github.com/sayanmondal2098/PaperHarbor.git
cd PaperHarbor
python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -r requirements.txt
.\.venv\Scripts\python.exe web.py

Choose a destination folder, upload a bibliography, select references, and start downloading. The default local address is http://127.0.0.1:8765/.

Import a reading list
.\.venv\Scripts\python.exe main.py `
  --export-file "references.bib" `
  --dest-folder "D:\Research\Papers" `
  --limit 10
Retry a previous report
.\.venv\Scripts\python.exe main.py `
  --retry-report "previous_report.csv" `
  --dest-folder "D:\Research\Papers"

The limit counts references evaluated, including skipped entries. Use --collection for a Zotero collection filter and --template to configure year, author, and title in filenames. Review CSV outcomes after a batch completes.

Installation instructions ↗

Data handling

Local files, visible outcomes.

Library access

The server binds to localhost. The SQLite reader locates common Zotero database paths or accepts a custom path. It queries a temporary copy and attempts to copy accompanying WAL and SHM files when present, then closes the connection and removes the snapshot.

Downloads stay in your chosen folder. PaperHarbor does not attach them to Zotero or update library records. Reference metadata is sent to discovery services during searches.

Practical boundaries

Access restrictions or missing metadata can prevent retrieval. Unresolved items retain follow-up links for manual review. Missing attachment detection relies on Zotero metadata, which may differ from files actually present on disk.

Sequential processing favors a traceable workflow over parallel throughput. An unsuccessful search does not establish that a paper is paywalled. Publisher access rules still apply.

Implementation: local Zotero reader ↗.