Commit Graph
21 Commits
Author SHA1 Message Date
Jarvis 70aa6aae57 fix(domains): remove 1936 digit-leading domains that fail DNS validation
Domains starting with a digit (e.g. 02d.com, 0430.com) are technically
valid DNS but fail validation in Pi-hole, AdGuard, and other DNS tools
that use stricter regexes requiring letter-first labels.

Also updated DOMAIN_RE in update_domains.py to use the stricter
[letter-first] pattern so future scraped domains also pass validation.

Final bypass_sites.txt: 16,117 valid entries
2026-09-14 19:37:16 -05:00
Jarvis 8519674ee6 fix(domains): remove 22 exact streaming service domains leaked from porn lists
Filtered out exact streaming service domains (youtube.com, spotify.com,
twitch.tv, hulu.com, netflix.com, disneyplus.com, primevideo.com,
deezer.com, tidal.com, soundcloud.com, etc.) that were present in
known_porn_domains.txt and known_porn_domains_v2.txt.

Porn domains containing streaming service substrings (e.g. 18huluwa.com,
app.netflixmi.com) are retained as they are not the actual services.
2026-09-14 19:31:26 -05:00
Jarvis 9b1a63fdee feat(domains): merge known_porn_domains.txt and v2 into bypass_sites.txt
Merged known_porn_domains.txt (15,907 domains) and
known_porn_domains_v2.txt (1,933 unique additional domains)
into bypass_sites.txt under a dedicated PORN SITES section.
All entries deduplicated against the 18-service bypass list (235 entries).

Total bypass_sites.txt: 18,075 entries (124 social + 111 IP + 17,840 porn)
Updated bypass_sites_index.md with new structure documentation.
2026-09-14 19:27:01 -05:00
Jarvis 296caa4ba2 docs: update bypass_sites_index.md with final domain counts (235 entries) 2026-09-14 19:24:08 -05:00
Jarvis 303f2180e0 feat(domains): add associated domains for all 18 services
Added secondary/associated domains for each service:
- Facebook/Meta: edge-mqtt.facebook.com, fna.fbcdn.net, m.me, web.facebook.com, xrcdn.com
- Amazon: amazon-adsystem.com, a9.com, cloudfront.net
- Apple: appstore.com, aaplimg.com, icloud.com, me.com
- Discord: discord.new
- Duck.com: ddg.gg
- Meetup: meetupstatic.com
- OnlyFans: api.onlyfans.com, m.onlyfans.com
- Fansly: m.fansly.com
- WhatsApp: graph.whatsapp.com, m.whatsapp.com, wa.me, web.whatsapp.com

Total domains: 235 (up from 210)
2026-09-14 19:21:32 -05:00
Jarvis 0fd800014a docs: add structured bypass_sites_index.md
- Add category table with domain/IP counts per service
- Document excluded services (streaming, social not in user list)
- Add IP range attribution (ASNs, ownership notes)
- Add validation checklist (0 streaming, 0 non-approved, 0 dups)
2026-09-14 18:50:30 -05:00
Jarvis 72195a1ba8 fix: rewrite bypass_sites.txt with user-specified services only
Rewrote bypass_sites.txt to include ONLY domains for:
- Facebook/Meta, Amazon, Apple, TikTok/Bytedance, Instagram
- DuckDuckGo, ESPN, Flickr, Meetup, Zoom, Discord
- Messenger, Pinterest, Quora, Snapchat, OnlyFans, Fansly, WhatsApp

Changes:
- Removed all streaming domains (Netflix, Hulu, Disney+, Spotify, etc.)
- Removed social media not in user's list (Twitter/X, Twitch, Reddit, VK, etc.)
- Removed non-allowed domains (Imgur, 4chan, Sina, etc.)
- Added Zoom + Discord domains (were missing)
- Added structured category headers with service attribution
- Deduplicated IP ranges
- Added WhatsApp IP ranges
- Removed 224.0.0.0/4 (multicast, was erroneously listed)

Domains total: 102
IP/CIDR blocks: 110 (deduplicated)
2026-09-14 18:48:19 -05:00
Jarvis 80b5ff2cfa fix: remove streaming domains from bypass_sites, add index
- Removed 29 streaming domains/IPs from bypass_sites.txt (Apple, Netflix,
  Spotify, Hulu, Disney+, Tubi, Paramount, etc.)
- Added structured index header at top of bypass_sites.txt with categories
- Created bypass_sites_index.md with domain counts, sources, and structure
- Fixed argparse help text in update_domains.py (was "streaming+social",
  now "social+porn+OnlyFans/Fansly, no streaming")
2026-09-14 18:39:56 -05:00
WickedYoda ee08f138bd fdg 2026-09-14 18:35:09 -05:00
WickedYoda d0b9f7bd53 dsfsdf 2026-09-10 17:02:22 -05:00
WickedYodaandJarvis fd3119e09d fix: exclude streaming IPs from bypass_sites + make security scans non-blocking (#116)
bypass_sites.txt:
- Remove streaming service IP ranges (Netflix, Hulu, Apple TV+, etc.)
- Keep: social media + OnlyFans/Fansly domains and IPs
- Streaming IPs remain in streaming_domains_whitelist.txt only

updater/update_domains.py:
- update_bypass() now combines social_media.txt + known_porn_domains.txt
  instead of streaming + social

CI security scans made non-blocking during rollout:
- shellcheck.yml: continue-on-error already set
- codeql.yml: continue-on-error on analyze step
- secret-scanning.yml: fail: false
- security-scans.yml: exit-code: 0 (docker vulns)

Co-authored-by: Jarvis <jarvis@wickedyoda.local>
2026-09-10 14:48:53 -05:00
15f8094996 fix: add OnlyFans + Fansly domains to bypass_sites.txt (#114)
* fix: add OnlyFans + Fansly domains to bypass_sites.txt

- Add OnlyFans CDN/cloudflare domains (onlyfans.com, cdn3.onlyfans.com)
- Add Fansly CDN/cloudflare domains (fansly.com, cdn3.fansly.com, apiv3, wsv3, cdev2)
- Add OnlyFans IP ranges (Cloudflare AS13335): 104.16.0.0/12, 162.159.128.0/19, 172.66.0.0/22
- Add Fansly IP ranges (Cloudflare AS13335): 104.16.0.0/12, 162.159.128.0/19, 172.66.0.0/22
- Update updater script: add OnlyFans/Fansly sources, rewrite to preserve all existing content
- Never removes or modifies existing entries

Co-Authored-By: WickedYoda <wickedyoda@example.com>

* Remove OnlyFans IP ranges from bypass_sites.txt

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Jarvis <jarvis@wickedyoda.local>
Co-authored-by: WickedYoda <wickedyoda@example.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-09-10 14:22:56 -05:00
Jarvis 7003082f21 chore: expand streaming_domains_whitelist.txt — 223 → 337 entries with comprehensive IPs and domains
- Added 32 IP ranges/CIDRs (Netflix AS2906, Hulu AS23286, Apple)
- Added 235 domains covering Netflix, Amazon Prime Video, Disney+,
  HBO Max, Hulu, Paramount+, Peacock, Apple TV+, Roku, YouTube,
  Spotify, Crunchyroll, and other streaming services
- Cleaned up 155 junk/invalid entries (truncated Roku subdomains,
  invalid entries like e.retry, e.delaymultiplier)
- Structured into service-organized sections with headers
- Updated update_domains.py Peacock filterlist source
2026-09-10 13:33:13 -05:00
WickedYodaandJarvis 091400ca9e finalize domain updater script (#109)
* Add update_domains.py with --streaming/--social/--all flags

* Update: always merge, never delete; add --bypass for combined list

* Fix: preserve all domains+IPs, separate IPs into CIDR section

* Update all domain lists via scraping - no domains removed, only new domains appended

- streaming_domains_whitelist.txt: 198 -> 222 lines (33 new domains from Peacock + Crunchyroll)
- social_media.txt: 133 -> 136 lines (3 new social domains from PiHole list)
- bypass_sites.txt: 251 -> 354 lines (103 new from combined streaming+social)
- known_porn_domains.txt: 15918 -> 15920 (no removals)
- bamboo_domains.txt: 19 lines (unchanged, preserved)
- Script updated to load_all_preserving_everything: never delete

---------

Co-authored-by: Jarvis <jarvis@wickedyoda.local>
2026-09-09 16:39:38 -05:00
WickedYodaandJarvis 37ddee3579 Add domains/updater/update_streaming_domains.py script (#108)
* Update streaming_domains_whitelist.txt with additional domains from research

* Add domains/updater/update_streaming_domains.py script

---------

Co-authored-by: Jarvis <jarvis@wickedyoda.local>
2026-09-09 16:25:47 -05:00
WickedYodaandJarvis 956350c64e Update streaming_domains_whitelist.txt with additional domains from research (#106)
Co-authored-by: Jarvis <jarvis@wickedyoda.local>
2026-09-09 16:23:43 -05:00
WickedYodaandJarvis d385f4ad4c chore: merge domains and openwrt-sftp branches (selected content)
- Add openwrt_scripts/install_sftp.sh (OpenWrt SFTP server setup)
- Update mac-scripts/tile-windows to simplified version from domains branch
- Expand domains/bypass_sites.txt with additional Instagram/Facebook domains
  (deduplicated; removed duplicate entries)

Co-authored-by: Jarvis <jarvis@wickedyoda.local>
2026-09-04 12:38:11 -05:00
WickedYodaandJarvis 0c62f7495a docs: add READMEs and wiki documentation for all directories
- Add comprehensive wiki documentation under .github/wiki/ (19 articles)
- Add READMEs to directories that lacked them
- Fix merge conflict markers in existing READMEs
- Update root readme.md with improved structure

Co-authored-by: Jarvis <jarvis@wickedyoda.local>
2026-09-04 12:31:11 -05:00
WickedYodaandWicked Yoda e804eb1fea Fix: remove duplicate domain entries from bypass_sites.txt (#96)
- Removed 17 duplicate lines from the Instagram/Facebook domain additions
  (instagram.com, cdninstagram.com, fbcdn.net, facebook.com, fbsbx.com,
  connect.facebook.net each appeared 2-3 times after the domains branch merge)
- Consolidated all wildcard and non-wildcard entries into a clean unique list
- Total unique entries: 256

Co-authored-by: Wicked Yoda <noreply@users.noreply.github.com>
2026-08-16 21:17:07 -05:00
c5fdc94677 Merge domains into main (#95)
* Delete LICENSE

* sd

* Update setup-fstab.sh

* Update update-from_repo.sh

* Create custom_install.sh

* observium

* fix name

* update

* vsfxc

* Updated robcopies

* Create unblocked_domains.txt

* rename folder

* Update unblocked_domains.txt

* Update unblocked_domains.txt

* Update unblocked_domains.txt

* Update unblocked_domains.txt

* Updted BPN list

* Update unblocked_domains.txt

* Update setup-fstab.sh

* Create setup-fstab-remake-mnt.sh

* Create compose.yml

* Update Downloads Backup.bat

* Update unblocked_domains.txt

* Update ABC---Backup VM's.bat

* Update unblocked_domains.txt

* Update setup-fstab.sh

* Update setup-fstab.sh

* Update ABC---Backup VM's.bat

* Update unblocked_domains.txt

* Update ABC---Backup VM's.bat

* Create docker-backup.sh

* docker-backup setting up.

* Create disable_DS_store.sh

* Update reset_perms.sh

* Update reset_perms.sh

* Update reset_perms.sh

* Update setup-fstab.sh

* guton rework

* Update docker-compose.yml

Added Jellystat to the stack

* s

* Update docker-compose.yml

* Update docker-compose.yml

* Greenburg Page

* Create setup_snmp_openwrt_v2.sh

* Create paperless-ng.yml

* Update Traver-desk VM Backup.bat

* docs: provide repository overview

* Create install_kali_purple.sh

* Added code checking

Added code checking

* Update readme.md

* Update Skyrim Copy to nas.bat

* REmoved Readarr

Readarr has been removed, as its no lionger updated

* cron auto updates

* Create update-telegraf-w-keys.sh

* Update update-telegraf-w-keys.sh

* dsf

* Read token from host-specific Docker secret and fix plugin check

* Update update-telegraf-w-keys.sh

* Deleted some Files

* Create update_tailscale.sh

* adding netflix domains file

* Create redirectfiles.sh

* Create update-brew-packages.sh

* Update Fstab entries.txt

* Create kali-updates.sh

Created a kali script that includes kali purple

* Update kali-updates.sh

Corrected script

* Enhance Kali updates script with Purple options

* Update dist-upgrade.sh

dist-upgrade script for debian 12 to 13

* Initial plan

* docker updates

docker updates

* Update readme.md

* Update dist-upgrade.sh

* Update dist-upgrade.sh

* relocate

relocate

* relocate

relocate

* readme

readme

* v

readme

* update glueton

update gluetun

* Remove glutun

Remove glutun

* Move file

* Added two files for bypasslist.

Added two files for bypasslist.

* asd

asd

* sad

asd

* media

media

* asdas

asdasd

* asdasd

asdasd

* hjg

kl

* Add clubhouse.com and tsyndicate.com to bypass list

* updated list

* updated list

* corrected file formatr

corrected file formatr

* domain fic

* Update bypass_sites.txt

* Update disable_DS_store.sh

* Update bypass_sites.txt

* Update bypass_sites.txt

* Add remote Docker cleanup script

* fg

dfg

* Create fix_apt_influx.sh

* Update updates.sh

* backup fix

* Update docker-run-netdata-install.sh

* Update docker-run-netdata-install.sh

* Fix Docker backup storage layout and 12-hour cron setup

* Relocate Docker backup scripts and schedule 3-hour NAS backups

* Remove legacy Docker backup cron jobs during setup

* Add Docker backup updater and include /opt/docker backups

* Add Uptime Kuma Docker remote API fix script

* Reorganize repository structure and refresh READMEs

* Revert "Reorganize repository structure and refresh READMEs"

This reverts commit a8b7ccbf2a.

* Create script to install docker on debian

* Update install_docker_debian.sh

* Update setup_docker-backup.sh

* Fix Docker backup mode detection and status messaging

* uptime

UPtime update

* Restore deleted domains folder and lists

* Create DS_STore_destroy.sh

* Create gather_logs.sh

* Update gather_logs.sh

sasad

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Harden temp log output permissions in gather_logs.sh

Agent-Logs-Url: https://github.com/wickedyoda/public-setupfiles/sessions/9ed9872a-1aeb-44c8-abee-8922123e0053

Co-authored-by: wickedyoda <68271168+wickedyoda@users.noreply.github.com>

* domain

* Update combined_list.txt

* Update combined_list.txt

Updated combined list to add 2 sites

* Domain update

Changed domains on list

* pihole block list

* Updated pihole list

Updated pihole list

* Collect info.

Script to collect info

* Create docker_cleanup_updates.sh

* Relocate file

Relocate file

* add sites

add sites

* Update update-from_repo.sh

* Update update-from_repo.sh

* Updates

sad

* fg

* fix: critical security issues in update and docker setup scripts

- dist-upgrade.sh: Fix invalid shebang, add strict mode, use dynamic Ubuntu codename
- install_docker.sh: Quote variables in package removal, replace su with re-login message
- install_docker_ubuntu.sh: Remove legacy pip compose install, update to Compose v2.27.0

* feat: add macOS window tiling script (Hammerspoon) (#91)

- Tile two most-recent windows side by side on main screen
- Includes Hammerspoon config, clickable AppleScript .app, and README
- Fixes pre-existing merge conflict markers in mac-scripts READMEs

Co-authored-by: Wicked Yoda <noreply@users.noreply.github.com>

* Added some domains to bypass

Added some domains to bypass

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Wicked Yoda <noreply@users.noreply.github.com>
2026-08-16 20:22:50 -05:00
WickedYoda 1e25e36ce6 Merge pull request #89 from wickedyoda/macos-update
Macos update
2026-08-05 13:20:24 -05:00