Windows PowerShell guide

Setup and run notes for Windows

Everything below is written for Windows 10/11 with PowerShell. The same commands work on Linux and macOS with python and ./run.sh-style equivalents.

Download project (.zip)

1Install Python

Download Python 3.10 or newer from python.org. During install, tick Add python.exe to PATH, then open a new PowerShell window so PATH picks it up.

Verify

python --version
# should print Python 3.10 or newer

2Get the project

Download and extract python-project.zip anywhere, then open a PowerShell window in that folder.

PowerShell

cd C:\Users\{you}\Downloads\python-project
ls   # you should see main.py and run.ps1 here

No dependencies: the tool uses only Python's standard library. There is nothing to pip install.

3Generate and edit your config

PowerShell

# write config.json (a safe starter)
.\run.ps1 -Init

# open it and edit with your own data
notepad config.json

Put your own export files somewhere and list them under exports (CSV or JSON).
Keep "fetch": enabled: false unless you truly have explicit permission to collect from a site.

4Run it (local-only by default)

PowerShell

.\run.ps1

# restrict categories / change the target on the fly
.\run.ps1 -Categories kargo telekom -Target 40

On success you get, under output\: records.csv, records.json, deficit_report.json, and run_log.json. Anything uncertain is in review\records_to_review.*.

5Run the tests

PowerShell

python -m unittest discover -s tests -v
# or
.\run_tests.ps1

Config reference

target_per_categoryTexts wanted per category - default 50. Multiplied across your chosen categories for the total.

categoriesWhich of the 20 canonical keys to collect. Defaults to all of them.

exportsLocal files to read. Each can be a path or an object with path, text_field, category_field, license.

sourcesNetwork sources. Only fetched when fetch.enabled is true AND rights_confirmed is true.

fetch.min_delay_secondsMinimum gap between requests to the same domain - default 2.0s.

fetch.max_requests_per_domainHard cap per domain per run - default 120. Raise only on resources you own.

fetch.respect_robotsObey robots.txt - default true.

pii.mask_thresholdhigh / medium / low. Medium (default) auto-masks emails, phones, IDs plus dictionary names, and flags the rest for review.

Example export entry

config.json

{
  "exports": [
    {
      "path": "data/sikayetlerim.csv",
      "text_field": "metin",
      "category_field": "kategori",
      "license": "own data",
      "rights_confirmed": true
    }
  ],
  "fetch": { "enabled": false }
}

How PII masking and review work

Confident identifiers - email, URL, phone, TC Kimlik No, IBAN, credit card, IP - are replaced with placeholders like [EPOSTA], [TELEFON], [TCN].
Known Turkish first names become [ISIM] and are flagged for a human to confirm.
Uncertain content (an unknown capitalized token, a possible address) is not silently deleted - it is flagged, and the record is routed to the review queue with its original preserved in a clearly-marked sensitive folder.

The clean dataset (output\records.*) never contains original text. The review\ folder does contain original text for uncertain cases - treat it as sensitive, keep it private, review it, then delete it.

The honest part

This tool is not legal advice and does not guarantee legal compliance. You are responsible for having the right to process, store, and redistribute the data you give it - including privacy and copyright rules in your jurisdiction.

It deliberately will not help you collect from a site without permission. For sources you are unsure about, prefer your own exports, openly licensed datasets, and documented APIs.

Deficit reporting is on purpose: if a category is short, the tool says so and ships the gap. Reach your target with more permitted data, or lower the target - never by inventing filler.

Back to the overview.