From 520c592bbd369df67ff426e357fb3dfbec2da5e1 Mon Sep 17 00:00:00 2001 From: Vincent Verbruggen Date: Sat, 7 Mar 2026 09:54:02 +0100 Subject: [PATCH] vault backup: 2026-03-07 09:54:02 --- .gitignore | 1 + .../Bathroom Blueprint copy.svg | 61 ++++++ .../Home improvement/Bathroom Blueprint.svg | 160 +++++++++++++++ Personal/Areas/Home improvement/Bathroom.md | 109 ++++++++++ Personal/Projects/Das Festival 2026.md | 194 ++++++++++++++++++ Personal/Projects/Dating/Breeze/Breeze.md | 4 + .../QLS-1 Server Crash Analysis 2026-03-04.md | 146 +++++++++++++ Work/Resources/Redis Cheatsheet.md | 188 +++++++++++++++++ atop_20260304 | Bin 0 -> 12665504 bytes 9 files changed, 863 insertions(+) create mode 100644 Personal/Areas/Home improvement/Bathroom Blueprint copy.svg create mode 100644 Personal/Areas/Home improvement/Bathroom Blueprint.svg create mode 100644 Personal/Areas/Home improvement/Bathroom.md create mode 100644 Personal/Projects/Das Festival 2026.md create mode 100644 Work/Projects/QLS-1 Server Crash Analysis 2026-03-04.md create mode 100644 Work/Resources/Redis Cheatsheet.md create mode 100644 atop_20260304 diff --git a/.gitignore b/.gitignore index aeb15ed..5712e49 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,4 @@ .claude .mcp.json /Personal/Areas/MCP-Servers +.DS_Store diff --git a/Personal/Areas/Home improvement/Bathroom Blueprint copy.svg b/Personal/Areas/Home improvement/Bathroom Blueprint copy.svg new file mode 100644 index 0000000..9c568ee --- /dev/null +++ b/Personal/Areas/Home improvement/Bathroom Blueprint copy.svg @@ -0,0 +1,61 @@ + + + + + + + + + + + + + + + + Bathroom + Blueprint + + + + + + + + + + + + + + + + + 1500 + + + + + + + + + + 2500 + + + + + \ No newline at end of file diff --git a/Personal/Areas/Home improvement/Bathroom Blueprint.svg b/Personal/Areas/Home improvement/Bathroom Blueprint.svg new file mode 100644 index 0000000..f22e1ab --- /dev/null +++ b/Personal/Areas/Home improvement/Bathroom Blueprint.svg @@ -0,0 +1,160 @@ + + + + + + Bathroom Blueprint (DRAFT) + Coordinates in mm — F/E corner = (0,0) — Y increases upward + + + + + + + + + + A + + + B + + + C + + + D + + + E + + + F + + + + + + + 2500 + + + + + + 2600 + + + + + + 1500 + + + + + + 1000 + + + + + + 1000 + + + + + + 1600 + + + + + + + + + + + + Bath + 1850x850 + + + + + + + + + + + + Shower + (~1000x1000) + + + + + + + + + + Sink + + + + + + + + + + + 600 + + + + + + + + + + 550 + + + + + + + + 0 + 500 + 1000mm + + + + + + Legend + + Wall + + Shower + + Water + + Radiator + + Door + + diff --git a/Personal/Areas/Home improvement/Bathroom.md b/Personal/Areas/Home improvement/Bathroom.md new file mode 100644 index 0000000..ff3b793 --- /dev/null +++ b/Personal/Areas/Home improvement/Bathroom.md @@ -0,0 +1,109 @@ +# Bathroom Renovation + +> **Started:** 2026-03-07 +> **Status:** Planning + +--- + +## Current State + +Describe what the bathroom looks like now: dimensions, fixtures, materials, condition. + +- **Dimensions:** _TODO (L x W in cm/m)_ +- **Ceiling height:** _TODO_ +- **Current flooring:** _TODO_ +- **Current wall finish:** _TODO_ +- **Fixtures:** + - Toilet: _TODO (type, condition)_ + - Sink/vanity: _TODO_ + - Shower / bathtub: _TODO_ + - Mirror/cabinet: _TODO_ +- **Plumbing location:** _TODO (which wall carries the supply/drain lines)_ +- **Ventilation:** _TODO (extractor fan?)_ +- **Electrical:** _TODO (outlets, lighting, heated floor)_ + +--- + +## Goals / Requirements + +What should the new bathroom achieve? + +- [ ] _TODO — e.g., walk-in shower instead of bathtub_ +- [ ] _TODO — e.g., double sink_ +- [ ] _TODO — e.g., heated floor_ +- [ ] _TODO — e.g., specific style/aesthetic_ + +--- + +## Blueprint + +![[Bathroom Blueprint.svg]] + +--- + +## Materials & Finishes + +| Element | Choice | Supplier / Link | Price | +|---------|--------|-----------------|-------| +| Floor tiles | | | | +| Wall tiles | | | | +| Shower screen | | | | +| Toilet | | | | +| Sink / vanity | | | | +| Faucets | | | | +| Mirror / cabinet | | | | +| Lighting | | | | +| Paint / other | | | | + +--- + +## Budget + +| Category | Estimated | Actual | +|----------|-----------|--------| +| Materials | | | +| Labour | | | +| Plumbing | | | +| Electrical | | | +| Unexpected / contingency (15%) | | | +| **Total** | | | + +--- + +## Contractor / DIY Plan + +- **DIY:** _list tasks you handle yourself_ +- **Professional:** _list tasks that need a contractor_ +- **Contacts:** + - Plumber: _name, phone_ + - Electrician: _name, phone_ + - Tiler: _name, phone_ + +--- + +## Timeline + +| Phase | Target Date | Done | +|-------|-------------|------| +| Planning & design | | | +| Demolition | | | +| Plumbing rough-in | | | +| Electrical rough-in | | | +| Waterproofing | | | +| Tiling | | | +| Fixture installation | | | +| Finishing touches | | | + +--- + +## Notes + +- + +--- + +## Reference / Inspiration + +Links, photos, or ideas collected. + +- diff --git a/Personal/Projects/Das Festival 2026.md b/Personal/Projects/Das Festival 2026.md new file mode 100644 index 0000000..9a8f7b8 --- /dev/null +++ b/Personal/Projects/Das Festival 2026.md @@ -0,0 +1,194 @@ +# Das Festival 2026 + +**Date:** Saturday, May 23, 2026, 12:00 - 23:00 +**Location:** Tuinen van West, Amsterdam +**Organizer:** [Komm Schon Alter](https://www.kommschonalter.de/events/komm-schon-alter---das-festival-2026) + +--- + +## Line-up + +### Andhim + +German duo (Simon Haehnel & Tobias Muller) known for their self-coined "super house" sound -- melodic yet deep, blending house and techno. Over 60 releases since 2010 and 100+ shows per year. + +- [SoundCloud](https://soundcloud.com/andhim) +- [Mixcloud](https://www.mixcloud.com/discover/andhim/) +- [Resident Advisor](https://ra.co/dj/andhim) + +**Recommended listens:** +- [Exclusive Mix for Time Warp](https://soundcloud.com/timewarp_official/andhim-exclusive-mix-for-time-warp) +- [True Underground ONE-65](https://soundcloud.com/trueundergroundone/andhim-tu65) + +--- + +### De Sluwe Vos + +Dutch DJ, producer, and label owner who combines house, techno, and hiphop with high-energy beats, driving basslines, and intricate percussion. Often incorporates old-school rave elements. + +- [SoundCloud](https://soundcloud.com/desluwevos) +- [Resident Advisor](https://ra.co/dj/desluwevos) +- [Bandcamp](https://desluwevos.bandcamp.com/album/now-i-understand) + +**Recommended listens:** +- [DJ Sets Playlist (SoundCloud)](https://soundcloud.com/desluwevos/sets/dj-sets) +- [The Boom Room 550](https://soundcloud.com/theboomroomofficial/550-the-boom-room-de-sluwe-vos) +- [The Boom Room 470](https://soundcloud.com/theboomroomofficial/470-the-boom-room-de-sluwe-vos) +- [Live @ Shelter 07.10.22](https://soundcloud.com/desluwevos/de-sluwe-vos-shelter-071022) + +--- + +### Eelke Kleijn (live) + +Dutch DJ and producer from Rotterdam. Blends progressive house, techno, and melodic electronic music with ambient and breakbeat elements. Runs the "DAYS like NIGHTS" radio show. Performing a live set at Das Festival. + +- [SoundCloud](https://soundcloud.com/eelke-kleijn) +- [Mixcloud (DAYS like NIGHTS)](https://www.mixcloud.com/eelkekleijn/) +- [Resident Advisor](https://ra.co/dj/eelkekleijn) + +**Recommended listens:** +- [Live at Mont Saint-Michel for Cercle](https://soundcloud.com/cerclelive/eelke-keijn-mont-saint-michel) +- [Live from Balance Croatia 2025](https://soundcloud.com/balance-series/eelke-kleijn-live-from-balance-croatia-2025) +- [The Boom Room 512 (3hrs)](https://soundcloud.com/theboomroomofficial/512-the-boom-room-eelke-kleijn) + +--- + +### Einmusik + +Berlin-based producer (Samuel Kindermann) specializing in melodic techno and deep house. Atmospheric, soulful sound with progressive influences. Runs his own label Einmusika. + +- [SoundCloud](https://soundcloud.com/einmusik) +- [Mixcloud](https://www.mixcloud.com/EINMUSIK/) +- [DJ Mixes & Livesets Playlist](https://soundcloud.com/einmusik/sets/einmusik-mixes) + +**Recommended listens:** +- [Mixes Playlist (SoundCloud)](https://soundcloud.com/einmusik/sets/einmusik-mixes) +- [Live DJ Set - Fusion Festival 2018](https://www.mixcloud.com/alfredo-raya/einmusik-live-dj-set-fusion-festival-2018-germany-27-06-2018/) + +--- + +### Huminal + +Dutch duo (Paul Deetman & Jelle Jansen) from Amsterdam. Progressive house, techno, and electronica with rich soundscapes, organic elements, and cinematic sound design. Blends analogue textures with crisp rhythmic forms. + +- [SoundCloud](https://soundcloud.com/huminalmusic) +- [Spotify](https://open.spotify.com/artist/0ifJHiwqO4YcVAd3vy8CEQ) +- [Resident Advisor](https://ra.co/dj/huminal) + +**Recommended listens:** +- [Flow Sessions 115 - Huminal (3.5hrs)](https://soundcloud.com/flowsessions/flow-sessions-115-huminal) + +--- + +### Mees Salome + +Dutch producer and DJ known for melodic electronic music blending emotive melodies with powerful grooves. Released on Filth on Acid, This Never Happened, and Armada. Founder of the Distant platform. + +- [SoundCloud](https://soundcloud.com/mees-salome) +- [Website](https://meessalome.com/) +- [Resident Advisor](https://ra.co/dj/meessalome) + +**Recommended listens:** +- [The Boom Room 439 (Resident Mix)](https://soundcloud.com/theboomroomofficial/439-the-boom-room-mees-salome) +- [The Boom Room 223 (Mixcloud)](https://www.mixcloud.com/theboomroom/223-the-boom-room-mees-salom%C3%A9/) + +--- + +### Miss Melera + +Amsterdam-based DJ and producer (Kim de Lange). Deep and progressive techno twisted into melodic, fresh compositions. Hosts a monthly podcast called "Colourizon." + +- [SoundCloud](https://soundcloud.com/missmelera) +- [Website](https://www.missmelera.com/) +- [Resident Advisor](https://ra.co/dj/missmelera) + +**Recommended listens:** +- [The Boom Room 138 (Mixcloud)](https://www.mixcloud.com/theboomroom/138-the-boom-room-miss-melera/) +- [1001Tracklists Overview](https://www.1001tracklists.com/dj/missmelera/index.html) + +--- + +### Nicky Elisabeth + +Dutch-Belgian DJ and producer. Romantic, eclectic deep house and progressive/melodic house with an energy-lifting aesthetic. + +- [SoundCloud](https://soundcloud.com/nicky-elisabeth) +- [Resident Advisor](https://ra.co/dj/nickyelisabeth) + +**Recommended listens:** +- [DHA FM Mix #463 (Deep House Amsterdam)](https://soundcloud.com/deep-house-amsterdam/nicky-elisabeth-dha-fm-mix-463) +- [DGTL Digital 2020 Set](https://soundcloud.com/dgtl-festival/nickyelisabeth-digital-dgtl-2020-1) + +--- + +### NTO (live) + +French producer (Anthony Favier) specializing in melodic and minimal techno. Atmospheric soundscapes with deep, purposeful beats. Often incorporates guitar and keyboards. Performing a live set at Das Festival. + +- [SoundCloud](https://soundcloud.com/anthony-favier-970568294) +- [WatchTheDJ (live sets)](https://watchthedj.com/djs/nto) +- [Instagram](https://www.instagram.com/ntomusic/) + +**Recommended listens:** +- [Lac de Soi - Cercle Stories (live in Swiss Alps)](https://soundcloud.com/cerclelive/ntolacdesoicerclestories) + +--- + +### Olivier Weiter + +Dutch DJ/producer deeply rooted in Amsterdam's underground scene. Known for marathon sets blending obscure house and techno. Founded the WEITER label and hosts monthly "WEITER Radio." + +- [SoundCloud](https://soundcloud.com/olivier-weiter) +- [Resident Advisor](https://ra.co/dj/olivierweiter) + +**Recommended listens:** +- [Yearmix 2023](https://soundcloud.com/olivier-weiter/olivier-weiter-yearmix-2023) +- [Yearmix 2022](https://soundcloud.com/olivier-weiter/olivier-weiter-yearmix-2022) + +--- + +### Rose Ringed + +Amsterdam-based DJ and producer. Melodic, emotive house and techno with warm, dreamy atmospheres and strong chord progressions. Founder of Closed Eyes Recordings. Collaborates regularly with Joris Voorn. + +- [SoundCloud](https://soundcloud.com/rose-ringed) +- [Resident Advisor](https://ra.co/dj/roseringed) + +**Recommended listens:** +- [Solomun Open Air Brussels 2023](https://soundcloud.com/rose-ringed/rose-ringed-dj-set-solomun-open-air-cinquantenaire-brussel) +- [De Zon Festival 2023 (Sunset Set)](https://soundcloud.com/rose-ringed/rose-ringed-dj-set-live-at-de-zon-festival-2023) +- [Diynamic OFF Sonar Barcelona 2023](https://soundcloud.com/rose-ringed/rose-ringed-dj-set-diynamic-off-sonar-barcelona-2023-monasterio) + +--- + +### Sandrien + +Amsterdam-based DJ and producer. Versatile techno ranging from deep and atmospheric to high-energy breaks and acid. Founded "Imprint," a monthly underground techno night. Regular at Dekmantel, Awakenings, and Loveland. + +- [SoundCloud](https://soundcloud.com/sandrien) +- [DJ Sets Playlist](https://soundcloud.com/sandrien/sets/dj-sets) +- [Website](https://www.dj-sandrien.nl/) +- [Resident Advisor](https://ra.co/dj/sandrien) + +**Recommended listens:** +- [DJ Sets Playlist (SoundCloud)](https://soundcloud.com/sandrien/sets/dj-sets) +- [Boiler Room Amsterdam](https://boilerroom.tv/recording/sandrien/) + +--- + +### Stephan Bodzin (live) + +German producer and classically-trained pianist. Melodic and minimal techno with deep, emotionally-driven soundscapes. One of the biggest names in melodic techno. Performing a live set at Das Festival. + +- [SoundCloud](https://soundcloud.com/stephanbodzin) +- [SoundCloud (Official)](https://soundcloud.com/stephanbodzinofficial) + +**Recommended listens:** +- [Live at Piz Gloria (Swiss Alps) for Cercle](https://soundcloud.com/cerclelive/stephan-bodzin-piz-gloria-for-cercle) +- [Piz Gloria on Mixcloud](https://www.mixcloud.com/Cercle_interlinked/stephan-bodzin-piz-gloria-for-cercle/) + +--- + +### Teyana + +*Could not find a verified SoundCloud profile or detailed online presence for this artist. They may be an emerging or local talent in the Amsterdam scene. Check the [Komm Schon Alter website](https://www.kommschonalter.de/events/komm-schon-alter---das-festival-2026) closer to the festival for updated artist info.* diff --git a/Personal/Projects/Dating/Breeze/Breeze.md b/Personal/Projects/Dating/Breeze/Breeze.md index c03e10f..dd6d9a9 100644 --- a/Personal/Projects/Dating/Breeze/Breeze.md +++ b/Personal/Projects/Dating/Breeze/Breeze.md @@ -29,6 +29,10 @@ When creating a Breeze profile, users provide: - Photos (minimum 2 required) - Personal bio - At least one Q&A prompt answer +## Potenial changes + +- Need more photos exspecially ones dressed nicly or in a suit. +- I need to show that i am looking for a bit of chaos, i am to good add buiulding routine. ## Q&A Prompts List diff --git a/Work/Projects/QLS-1 Server Crash Analysis 2026-03-04.md b/Work/Projects/QLS-1 Server Crash Analysis 2026-03-04.md new file mode 100644 index 0000000..9e72619 --- /dev/null +++ b/Work/Projects/QLS-1 Server Crash Analysis 2026-03-04.md @@ -0,0 +1,146 @@ +# QLS-1 Server Crash Analysis - 2026-03-04 + +## Server Specs + +- **Instance:** AWS EC2 (aarch64) +- **CPUs:** 4 +- **RAM:** 15.4 GB +- **Swap:** 1 GB +- **OS:** Ubuntu 22.04, kernel 6.8.0-1029-aws + +## Summary + +The server ran out of memory during a deployment and became completely unresponsive, requiring a reboot around 09:55 UTC. A deployment (unzip + node build) pushed an already critically overloaded system over the edge. + +However, the real problem is that a single-site server with a few background processes should not be consuming 14 GB of RAM during normal operation. The deployment was the trigger, but not the underlying cause. Memory usage needs to be profiled and reduced. + +## Timeline (UTC) + +| Time | Event | +|---|---| +| 00:00 - 09:36 | Normal operation. Memory already dangerously tight: ~14 GB used, ~700 MB free, swap fully used (1024 MB). | +| **09:37** | Memory pressure intensifies. Used = 14758 MB, free = 291 MB. Cache shrinking. | +| **09:39** | `sessionclean` cron + `find`/`sed` spawn 59 processes each. Used = 15168 MB, free = **180 MB**. | +| **09:41** | **Deployment starts:** 233 `unzip` + 233 `rm` processes appear. Free = **119 MB**. | +| **09:42** | Cache evicted (172 MB), buffers = 9 MB. **1,181 allocation stalls.** Page scanning = 886,726. | +| **09:43** | `node`/`esbuild` build starts. **Redis forks for persistence.** D-state jumps to **121**. Alloc stalls = **4.4 million**. System grinding to a halt. | +| **09:44 - 09:54** | **Death spiral.** Free ~120 MB, cache = 0, buffers = 0. D-state peaks at **196**. Alloc stalls ~20 million/min. Cron jobs pile up (15 to 45). System frozen. | +| **09:55** | **Reboot.** Used drops to 262 MB, free = 15,251 MB. Swap = 0. | +| 09:56+ | Services restart. Normal operation resumes. | + +## Key Numbers at Crash Point (09:43) + +- **RAM:** 15,525 / 15,768 MB used (98.5%) +- **Swap:** 1024 / 1024 MB used (100%) +- **Page cache:** 112 MB (down from ~2 GB) +- **Buffers:** 0 MB +- **D-state processes:** 121 (normally ~60) +- **Alloc stalls:** 4.4 million / minute + +## The Actual Problem: Per-Process Memory Breakdown + +The deployment was the straw that broke the camel's back. The real issue is baseline memory consumption. This server hosts a single site with a few background processes, yet it was consuming ~14 GB of 15.4 GB available during normal operation with swap completely exhausted. + +The atop log contains per-process resident memory (RSS) data. Here is what was running: + +### Memory by process group at 09:30 UTC (normal operation, pre-crash) + +| Process | Count | Threads | Total RSS | Total PSS | Swap | Notes | +|---|---|---|---|---|---|---| +| **redis-server** | 1 | 7 | **28,398 MB** | 28,390 MB | 448 MB | Single instance using 28 GB RSS | +| **php8.4** (CLI) | 27 | 27 | **12,835 MB** | 10,798 MB | 534 MB | Queue workers / artisan commands | +| mysqld | 1 | 55 | 8,096 MB* | ~1,500 MB | 2,551 MB | *RSS inflated by mmap'd InnoDB files | +| **php-fpm8.4** | 10 | 10 | 2,153 MB | 567 MB | 0 MB | FPM pool (PSS=567 MB shared) | +| **php** (CLI) | 1 | 1 | 1,756 MB | 1,638 MB | 0 MB | Single CLI process | +| **php-fpm8.3** | 15 | 15 | 131 MB | 5 MB | 489 MB | Old pool, mostly swapped out | +| **php-fpm8.2** | 7 | 7 | 69 MB | 11 MB | 221 MB | Old pool, mostly swapped out | +| nginx | 5 | 5 | 205 MB | 63 MB | 6 MB | | +| supervisord | 1 | 1 | 50 MB | 44 MB | 41 MB | | +| snapd | 1 | 12 | 79 MB | 78 MB | 8 MB | | +| amazon-ssm-agent | 1 | 11 | 46 MB | 46 MB | 4 MB | | +| squid | 2 | 2 | 43 MB | 21 MB | 46 MB | | +| meilisearch | 1 | 13 | 39 MB | 33 MB | 27 MB | | +| Everything else | ~55 | - | ~300 MB | - | - | | + +**Note on RSS vs PSS:** RSS counts shared memory and memory-mapped files in every process that maps them, leading to overcounting. PSS (Proportional Set Size) divides shared pages among users. For PHP-FPM workers, PSS is much lower because workers share the same code pages. For MySQL, RSS is heavily inflated by memory-mapped InnoDB data files that are backed by disk -- actual private memory usage is ~1.5 GB (8%), not the 8 GB RSS reported by atop. + +### The three big offenders + +#### 1. Redis: 28 GB RSS (PID 781977) + +This is by far the largest consumer. A single redis-server process with 28 GB RSS on a 15.4 GB server means it has a working set far larger than physical RAM. The kernel is constantly swapping redis pages in and out, contributing to I/O pressure and the high D-state count even during normal operation (~60 D-state processes at all times). + +This needs immediate investigation: +- What is stored in Redis? Is it being used as a primary data store instead of a cache? +- What is `maxmemory` set to? If unset, Redis will grow without limit. +- What eviction policy is configured? (`maxmemory-policy`) +- Is `save` / RDB persistence enabled? Forking for persistence doubles memory briefly. + +#### 2. PHP 8.4 CLI workers: 27 processes, 10.8 GB PSS + +These are likely Laravel Horizon / queue workers. Individual workers range from 220 MB to 2,867 MB RSS. Two processes stand out: +- PID 165508 (php8.4): 2,867 MB RSS at 09:30, growing to 4,584 MB by 09:41 - a clear memory leak +- PID 165699 (php): 1,756 MB RSS at 09:30, growing to 4,569 MB by 09:43 - same pattern + +These workers appear to have memory leaks. They should be configured to restart after processing N jobs (`--max-jobs`) or after exceeding a memory limit (`--memory`). + +#### 3. MySQL: ~1.5 GB actual usage + +The 8 GB RSS reported by atop is misleading -- it includes memory-mapped InnoDB data files backed by disk. Actual private memory usage is ~1.5 GB (8% of RAM as seen in `top`). MySQL is not a significant contributor to this crash. + +### Memory progression leading to crash + +| Time | Redis RSS | PHP8.4 CLI RSS | MySQL RSS | Free RAM | Event | +|---|---|---|---|---|---| +| 09:30 | 28,398 MB | 12,835 MB | 8,096 MB | 766 MB | Normal | +| 09:38 | 28,514 MB | 14,648 MB | 8,082 MB | 291 MB | PHP workers growing | +| 09:41 | 28,573 MB | 15,279 MB | 8,077 MB | 119 MB | Deployment starts | +| 09:43 | 57,464 MB* | 12,701 MB | 8,030 MB | 129 MB | Redis forked, system dead | + +*At 09:43, a second redis-server process appeared (PID 173622, 28,861 MB RSS). This is Redis forking for BGSAVE/BGREWRITEAOF (RDB/AOF persistence). On a system with no free memory, forking a 28 GB process is fatal because of copy-on-write page faults causing massive memory demand. + +**This is the kill shot: Redis persistence fork on an already OOM system.** + +## Immediate Actions + +### Priority 1: Redis (the root cause of the memory crisis) + +1. **Set `maxmemory`** to a reasonable limit (e.g., 2-4 GB). If Redis holds more data than that, it needs to be moved to a dedicated instance or the data model needs rethinking. +2. **Set `maxmemory-policy allkeys-lru`** to enable eviction. +3. **Disable or schedule RDB persistence** (`save ""`) to prevent the fork that killed the server. If persistence is needed, use AOF with `aof-use-rdb-preamble yes` and schedule `BGREWRITEAOF` during low-traffic windows. +4. **Audit what is stored in Redis.** Run `redis-cli --bigkeys` and `redis-cli INFO memory` to understand what is consuming 28 GB. + +### Priority 2: PHP CLI workers (memory leaks) + +1. **Add `--max-jobs=500` and `--memory=256`** to queue worker commands to force periodic restarts. +2. **Investigate PIDs 165508 and 165699** specifically. These grew from ~2-3 GB to ~4.5 GB over 13 minutes. That rate of growth suggests either a leak or processing abnormally large jobs. +3. **Reduce worker count.** 27 CLI processes is excessive if they each consume 300-500 MB. Start with 5-8 workers and measure throughput. + +### Priority 3: MySQL (low priority) + +MySQL actual memory usage is ~1.5 GB, not the 8 GB RSS reported by atop. Not a significant contributor to this crash. No action needed unless memory remains tight after fixing Redis and PHP workers. + +### Priority 4: Clean up legacy PHP pools + +- **php-fpm8.3** (15 workers, 489 MB swap) and **php-fpm8.2** (7 workers, 221 MB swap) are almost entirely swapped out, meaning they are not actively used but still consuming swap space and incurring I/O when occasionally touched. Remove them if no sites use those versions. + +## Deployment Hardening + +Even after fixing the baseline, deployments should not be able to crash the server: + +1. Serialize unzip/rm operations instead of 233 parallel processes. +2. Cap node build memory with `NODE_OPTIONS=--max-old-space-size=512`. +3. Add a pre-deploy memory check: abort if free memory is below a threshold. +4. Protect critical services with `OOMScoreAdjust=-900` in their systemd units. +5. **Disable Redis persistence during deployments** or ensure `BGSAVE` cannot trigger concurrently. + +## Longer Term + +- **Redis belongs on a separate instance** (or use ElastiCache) if it genuinely needs 28 GB. Running it on the same 15 GB server as MySQL and PHP is fundamentally unsound. +- Increasing swap from 1 GB to 2-4 GB buys time but does not fix the root cause. +- Upgrading to a larger instance is treating the symptom. The workload should fit comfortably in 16 GB with proper limits on Redis and MySQL. +- Set up memory monitoring and alerting (CloudWatch, Datadog, etc.) so this does not go unnoticed until it crashes. + +## Data Source + +Analysis performed by parsing the binary atop log file `atop_20260304` using Python + atoparser. The file covers 00:00:02 - 09:58:04 UTC with ~1 minute sampling intervals (599 records). Both system-level (memory, CPU, disk) and per-process (RSS, PSS, swap, state) data was extracted. diff --git a/Work/Resources/Redis Cheatsheet.md b/Work/Resources/Redis Cheatsheet.md new file mode 100644 index 0000000..dcfc23b --- /dev/null +++ b/Work/Resources/Redis Cheatsheet.md @@ -0,0 +1,188 @@ +# Redis Cheatsheet + +## Memory & Diagnostics + +```bash +# Overall memory stats +redis-cli INFO memory + +# Human-readable used memory +redis-cli INFO memory | grep used_memory_human + +# Total number of keys +redis-cli DBSIZE + +# Memory usage of a specific key (in bytes) +redis-cli MEMORY USAGE "some:key" + +# Find biggest keys per type (safe, non-blocking) +redis-cli --bigkeys + +# Top 30 keys by memory usage (scans first 500 keys) +redis-cli --scan --pattern "*" | head -500 | while read key; do + bytes=$(redis-cli MEMORY USAGE "$key" 2>/dev/null) + echo "$bytes $key" +done | sort -rn | head -30 + +# Full scan (slow on large datasets) +redis-cli --scan | while read key; do + bytes=$(redis-cli MEMORY USAGE "$key" 2>/dev/null) + echo "$bytes $key" +done | sort -rn | head -50 + +# Memory doctor -- quick health check +redis-cli MEMORY DOCTOR +``` + +## Key Inspection + +```bash +# Count keys matching a pattern +redis-cli --scan --pattern "cache:*" | wc -l +redis-cli --scan --pattern "horizon:*" | wc -l +redis-cli --scan --pattern "queues:*" | wc -l + +# Check key type +redis-cli TYPE "some:key" + +# Check TTL (-1 = no expiry, -2 = key doesn't exist) +redis-cli TTL "some:key" + +# Find keys with no expiry (sample first 100) +redis-cli --scan --pattern "*" | head -100 | while read key; do + ttl=$(redis-cli TTL "$key") + if [ "$ttl" = "-1" ]; then echo "NO EXPIRY: $key ($(redis-cli MEMORY USAGE "$key") bytes)"; fi +done + +# List all key prefixes and their counts +redis-cli --scan | sed 's/:[^:]*$//' | sort | uniq -c | sort -rn | head -30 +``` + +## Streams + +```bash +# Stream length +redis-cli XLEN "stream:key" + +# Stream info (memory, first/last entry, etc.) +redis-cli XINFO STREAM "stream:key" + +# Trim stream to max N entries +redis-cli XTRIM "stream:key" MAXLEN 10000 + +# Delete a stream entirely +redis-cli DEL "stream:key" + +# Read last 5 entries +redis-cli XREVRANGE "stream:key" + - COUNT 5 +``` + +## Hashes, Lists, Sets, Sorted Sets + +```bash +# Hash: field count and sample fields +redis-cli HLEN "hash:key" +redis-cli HSCAN "hash:key" 0 COUNT 10 + +# List: length and sample +redis-cli LLEN "list:key" +redis-cli LRANGE "list:key" 0 9 + +# Set: member count and sample +redis-cli SCARD "set:key" +redis-cli SSCAN "set:key" 0 COUNT 10 + +# Sorted set: member count and top 10 by score +redis-cli ZCARD "zset:key" +redis-cli ZREVRANGE "zset:key" 0 9 WITHSCORES +``` + +## Memory Limits & Eviction + +```bash +# Set memory limit (runtime, no restart) +redis-cli CONFIG SET maxmemory 2gb + +# Set eviction policy +redis-cli CONFIG SET maxmemory-policy volatile-lru + +# Persist config changes to redis.conf +redis-cli CONFIG REWRITE + +# Check current settings +redis-cli CONFIG GET maxmemory +redis-cli CONFIG GET maxmemory-policy +redis-cli CONFIG GET save +``` + +### Eviction policies + +| Policy | Behavior | +|---|---| +| `noeviction` | Refuse writes when full (default -- dangerous) | +| `volatile-lru` | Evict LRU keys that have a TTL set | +| `allkeys-lru` | Evict any LRU key | +| `volatile-ttl` | Evict keys with shortest TTL first | +| `volatile-random` | Evict random keys that have a TTL | +| `allkeys-random` | Evict any random key | + +## Persistence + +```bash +# Disable RDB snapshots (prevents fork that doubles memory) +redis-cli CONFIG SET save "" + +# Trigger manual save (blocks until done) +redis-cli SAVE + +# Trigger background save (forks -- dangerous if low memory) +redis-cli BGSAVE + +# Check last save time +redis-cli LASTSAVE + +# Check if background save is in progress +redis-cli INFO persistence | grep rdb_bgsave_in_progress +``` + +## Bulk Operations + +```bash +# Delete all keys matching a pattern (use with care) +redis-cli --scan --pattern "prefix:*" | xargs -L 100 redis-cli DEL + +# Count + size of keys matching a pattern +redis-cli --scan --pattern "horizon:*" | while read key; do + redis-cli MEMORY USAGE "$key" +done | awk '{s+=$1} END {printf "Total: %.2f MB\n", s/1024/1024}' + +# Flush current database (DESTRUCTIVE) +redis-cli FLUSHDB + +# Flush all databases (DESTRUCTIVE) +redis-cli FLUSHALL +``` + +## Monitoring + +```bash +# Watch commands in real time (Ctrl+C to stop) +redis-cli MONITOR + +# Slow log -- queries that took longer than threshold +redis-cli SLOWLOG GET 10 + +# Connected clients +redis-cli INFO clients | grep connected_clients + +# Stats since startup +redis-cli INFO stats +``` + +## Config File Locations + +``` +/etc/redis/redis.conf # Debian/Ubuntu default +/etc/redis.conf # RHEL/CentOS +/etc/redis/6379.conf # Multi-instance setups +``` diff --git a/atop_20260304 b/atop_20260304 new file mode 100644 index 0000000000000000000000000000000000000000..f0761179e13e12b2f72c958182268bfb8a5142fc GIT binary patch literal 12665504 zcmeEtWmJ@J*Ds1lqku>aAPfkCfb`G`$RJ1zjg)jqcZwn{%+N^3NH+o^-QC?OE!{D2 zZt(v+&w4+c^`3LSowLrY#p0f8uH4to-`@95?ba2hHyZGnzw;i=@aEy)pHl<@niM)3 z+FLU#N0)!M{vSNR-YT(j{2z?__Yht-2-^!*ju)K#ta{G&|8D+Ye*klFeB|V0d%?}d z!K$L~XyxGeL`hcRiL$BVlUFu&PdK@saJ~@W;THfNRFuVmFPuF8b#gsDI|EZ*?*H2O zzw?dB%G%24+5gUn|9RhsXlOUvyAS+f{m*;d{&LqT)9mKqU*xC8L_?dEMnf}v;e03Z z4jLLJ@b|C(H*>K8-$idnqM-rH{I?&MgstHZrXxlm%-zHvuJOI(Mqu%3 zN-J~BO>M1Sy%~K?#_^1iuhYbNZi2nDco|#>->=`7_mY!9wSF8(y*kqq6o5`?$YjXN zmwx}4_DUz`*;2B+d=H2qhz50dZ#u8T5)UT0h{w_Qz!FN?2b(uM12bYwqiF47Nk% zpzHO@YLEHydrc*u2?-{%6Z~>s_37d?abo_N5KFmodWs0z6{3-`dSGARx|{uCHDo!q z@1TiCTx>KNUiaMo?te@F|KH(%l>+tG8q0%=p-nhH#Ni2)J2cYexUnbt7olezF4vc5 zr_<}bd*|bksH!xrtLfoH*r-J>_64O$l;5;N3Da-)Dl4qB^Iq@Ji*g*|(Ei zBce*P)p?R9l+tRkKR5~Ph7mkN;j$u8GBE`jah;Rkh@GstOiQklD!iaRlVl-?K<9DK z;hR#M0(WSe#^8J-3vy8_)N|oNm?(B4K{qb@XOXXCE6>1*FJG4(W$ur~Etfj;zGGac zwaK@0x<8F)b+1u5!AA>LJ&#OVC(;^2ci`(pb_QcQw9xd*g!ZpoDE_(V&JtyvKdZK= zGGFq|{5G!ZSu>fo(3$w`nC+OP4{du-5ytI>TLY%32KOb@p_450&*4@$j3l}g=? zzCkyyoLBMG6=nu^Mq5m`&PnnWi2u7Q`NSem7iGzU87-mls}z#wPh#x@d)Az^!>>rfDa)3!y_T#PiPUXV$sI=3Bh~3r_a`^cJtF0^JJj>$Eh5%=UIu z2R6yAWi{PDfLEb3Nz--30%|HAgM6J2KJF;(Z*iT(X#LRhd{ZiN(Y{FaV+riIIT&sm zujYq};XSNfUhvwnydjm1?H(PyA(G&VH5}9Lt1U-|vGIQ4DyLV=|s{J%K$qFT`P1WiC%A zIjY;K?K7Nn=KOKdTlJ$#ZxyxC6>-ubD9F{}v0uBxwF6UEP*6~>f2zz@{S*}DS>R)B zHtjGiQaCIdUFI$v;>47v-eKIN+fr-Op~wH*)Bh7Q-!TN zQTTM}QdiRKg7kT7c?Dzn&nV8>epp7@-QziubDlDZY1)Z}jeFU<(-rHDVIwW)Ki>@W zxiQpQjj?CsdOb!hT8z#aJUwJ(@{b8sI{MVSFZi-g^2Nhl-ck%P)?B2WveiS-dOH*J zyA)hq)TFtcTh7!wuNNjSl?N(&;`ck$Gn@5yFfTXhBeq~K8Jq94)X@aZNw3v5^tF|F z*T)_DWi?e-vq)>5tfIPS<0{D0c*C-|8kf{s2LCV=Ze zd;S>EH<7Y%V}jq1b_tC{ls7Hwf@8V_oD@xRb}F0+2hO3% zdmgr*Zs-EmJ^C{#Kq*fE}i!K!}je60xjxEcH(-zzPwsl>;kMN566WsiU z2xP{R_*7Nnc@*Vk@E88Puc3Lp11|v5^M8Rk_a~8EvOxx;HI|e$9G703CG_6|mRSf5 zwpEVa#fEc7ZSBpf3C;3_aAmOP+H>C}lCL5dpuZWgB{15jaoYCxDhH&bT9@!DQZHMRdOFM{r&&LIYH=U`usi!UjaDbrEkFJ2?x%ZKRs;nFLWnC_Dr)IKLsQ_+lVe^F^aK zST>f#>N)-gVmV0FY*YCGWPsl^KBELe!rH$;V=PG7Fj9f6)RVi=7=e-JH zyMgCYO2#eZ>Z>&Qp8O%iaZdnJP!q`~-1$q2n`JVl<6E(1Lr3`I(12y`jJ@#&TaquY z5hViaKp7Lt-G2bf$Pv?$Be<~O;ek=$<<%=xZs`(U9!R^&ZrUJ}+icyaIR6-?3S0Cl?`<|no`!)y2uK;-EmC2(e1^3|+? zsYQ#&AcnB{c7Ytf8zJo zcydcVPun*@te=3nrT;D%A>kRR#OebyF#l~BNfYVJ?!D<@{Xx!WMF6Sit1XE`E`~9#6hHY0Ei2rWXeV?%_R)vV^x{1KVi{4YpCHZRPr|T+UCy zzUat5G68Im{+|Z=4P=5j_J4*n-Y)&wFaXSvM4MhBGOO8KW1o}$pAqJSM=&`@wtq%2 ztk7u4bUh(olnOSH%C@lQR;~8$$?gRR_ji0xn9?wmf8>TF|E#$2@Mip-e~*_{m+7J< zU&vDYX9DQW1Z)5?{+R$KWfQ5SeeXZVW^8dEPuS3x@`Dg9kt;G!f%1t_z!@<9HREj4 zJDHf5oXufmeW8jIE2d$lW6R1hV*fPIZCkSP{O8v-$_AKR8PR`eXdM91aQTy=2Mvcm z*P0oiH5kIk0o8#zg%j|IarTDQjWA9D!eDqG3jLbb8%7*H-|i9U$(j!)rEds3BLZna zId31f1U9qC!)@UnI)@jJLF+QjIdy1Uz4J%twiDl`X`4`C+(sv5zCLtz?Ao9dfL|JKaE%wFJU=!48rykRZ( z1oIoADfrynIglf%_KT{w})WgK@06_71G{ODOO!VZ~>yT=WqET#-fpMgOrcjEGDTR*>Y5~ZuL zzsEbr251n43s>yKf+BPf`ElUrIo!b*O`U1G_|FHiz?6qL^zf4+0_)*{j)A^=Q0>Sb zry0>?eFioz!0Z)@FV44kZyq!=i^*{G`c8 z(A{XUHjJ4UW^`>6-4VCC0 zmM#A%QmyZK1+eb5H)@U&o92uE}y_VPnWTB!nAl}7(v!!>o$f6Kk;Kp#2mF2u`T8$ ziDIVs5w_eA$ z-=0S7wEa9Fss%IIbx*xDjXZ3G|9tgDLiFR2M;;WpG3^d7qxyHa*9b) z%-}?Om8W;Y!H@QQs6KX3VY#y5o^C#bD@r-JHfE)-?sjl#+eLskLQ-~}xS9Ol2`{`f zUD;4iS2MO(^94C#7v?`#%^Fy+z~<)Zc^=}qWnVf)mo#p|6sz*Nb+@}@&_(qaL+pcb zw3unw0_K8HDFd<0!q*zyuhKN43IRIL(mXnyZ*MBrU`EfJ2m0O5R;qe?~6J* zQ(J70s88Ec8zr7Iu^jWPso?+@T3oM6ThQid_YXLfBUf@klj-0;U6x<~p_xw@r)wDcbI1FvFZoLX|Rb{Q= zqsVmNkJ};zh3R{~P5LtWEpd165Fx4kgU4*(U6j#?^VC{(<3PDnch~#*0NnG?!}=U_-{xzw?>V&bK4%o z_7Kt8aWz2fjb!88XN_TOm5$yX%ot)x6hMtQ_j0@G~YUC6neFztxs)j00gXXLBS>;7Xf9ryO#Ga$v+DQ>u4HrJlIVeg;X(V*;v=(w%8{odG@U`ws#2!SPyk&rdwju0YH7i-t z)xfYjzr1lc;1AMM%C&AH6?3#tv}-XPbql7xA`R zo!^ywR~V|>5~>uzSuyi;&?J#-U7B^F|Efr%&Ax$T>olL9T0>KgFaqQu!Tjb9Qx@^O zh(L}o-lf^(?fJj4v{!B`c2E3!kS??@UY&xE$~V7bMb-xlH%8ROI-g5Kdp$K~CXwkZ zFK}BTSJ$}wSg7{&g>K{>ZNbo9mV>7v6_0*Od>C6|nRcr!uOmtd@<`?@E~H?rnc8@x zuG~s#N`OHJ4;WNYIER|U7g5O5xL1i#DJTBLw}vtacf6FIT4ien9C5%FCp+nd4!_N| z+tx;`3eEa=7Yf14H;%rjAIYouR^0o8b2?2>VdINicKsw$4RZb@6B)yqi5h{Rj(=P| z>q?`E;(C-qmzvElV7Wf=l|eJYP*d?eZEV^0BK4RwZL!$%%}1 zXT?=c=cetU_*Z*}A_~UgawUZB>D#Eq3)w7Hn$cw}+DA?@yqVvr&HbGcv7kb|MWFN& zhT>|sEba02Xl+=b7HQCVy2fNOZ152lPnT1AR+p-(k_JrsoemKTeN~wngd#QzM*1cK&O33#t4KK?HFL6gJS&zKHbVIl4Os{A1O3~$2_|B4dZC)#OORYxh zSgZmKIr(|M8ckU8K$%ER{85O*f-ks@T z!1z*Mbr0#TJKg6{HnzRAddx`t)knBZolaHTfbwRo2{~~XP64PioVxTA2YQ{|x-Qdt z9`IAUO%vPbz$6Sieeq>%OU_toft-c}_6 zRh`NPvM|?HpnJTaDTKQ*vni~hH6ixrFl%#XSP7d!<&tvxz}VB7U*DWXYX@>rEqwSb z<%g;sMpQ3Sf3~J>IX7G~_#)x6XBsplqw|L(2A<6$1zd$hohyjeN9k_nku8+&gd#~! zGYGE}WCzGUeb}E_0I+Ow0bL=k7^~J+qsKHpJs1Zy+GEK3ndq@!xwr-Nqw~J?9~WJj z7SXSWm@nR!jM{RV*}OWdQH`@glB-SOTX(48D?%mDz$j#H+n}p2c#L7(DiN_IFE?^5GvDV7m&CCDc^sH2@^07xcO+_ZDyF=%0vnn8DVZQSM;R#oA-fp@yOZr9zXA+^5e6O-4uZL5- z%m+k=46Y+n!JYktMr#3_4ie-7X?RpHKRA;(^l$U>T7aE{1iwHUhzbUWGeMz|NO0!> z;ooKqfwa3+u+Q+Umr&&tFmpd4$yxx8gT&wFKzNn}R5=aIJU|$=79i~)!61-U;&&d) z5aCwygdSk2+L$0{=$T^F@`&}q{m2W2jUS5XbN7ZsyWGq#tiPaVHik(5!e=pl%fkaH z-lp<@XI9>-kiz=BE#BE;OJE=CL;KqIN{7{^$0$=FU#*(X{SAJS_D;J9^zud*Wz(NL z_@l==-{fhW4Q=B{hh>d?!0BERu+*lok*F=#Wv<@mX>uBcCDoHX?Z5%%q(O=1pOD7M zP2wST=