Updates

Show file flags on the dashboard: the errors and warnings Blot now stores per file (#2673). Each code, with its message, its source and the detail it carries, is in the dictionary in app/models/file/flags.js. storage/folder's stat and folder listings give a file's flags, plus placeholder: true for a placeholder, and storage/folder.flagged(blogID) walks every flagged file in a blog. Ideas: a badge and the message beside each flagged file in the folder viewer, the detail on its file page, and a list of every flagged file per site.

Flag folder posts' sources: app/sync/update/set.js sets no build, assets or references flags for a + folder post, since the build has no one file to put them on. app/build/multiple.js now sets err.sourcePath on a failed source (#2677). Do the same for the flags each source's build reports, and set them on that source.

Noticed while flagging the existing placeholders, 12 Nov 2026 (#2675). The resync failed for a few blogs, every time:

• Two Dropbox blogs answer 401: their access was revoked, but they still count as connected. Find out whether the hourly validation and the dashboard tell the owner, and if not, mark the blog as needing to reconnect.
• Three Dropbox blogs fail the whole resync on a 409 for one path (one was a .url file). See the Dropbox 409 entry below. A refused file should not stop the rest of a reset.
• One Dropbox blog never finished its initial transfer, so its reset refuses to run. Find out how long it has been stuck and whether the owner was told.
• One iCloud blog: the macserver's /download answers 500 for three files, every time. Check the macserver logs for why.

Lift the file name length limit rather than trim names. The check behind it, exceedsFilesystemPathLimits in app/clients/dropbox/util/constants.js, dates from when blog folders lived on ext4: 255 bytes per name, 4096 per path. Folders are in S3 now:

• Directory names and full paths no longer touch a local filesystem. S3's own limit is 1024 bytes for a key.
• The one place that still does is the content cache on the instance SSD (app/storage/folder/cache.js). Its directory is a hash, but the file inside keeps the original basename, presumably so pandoc and sharp can see the extension. Shorten that local basename, keeping the extension, and a long name stops mattering.

Unescaped tag in tag-page titles: when /tagged/<tag> matches no entries, app/blog/render/listingViews.js sets res.locals.tag to the raw req.params.tag, and 18 of the 21 templates put it into the title and description partials with {{{tag}}} (in package.json), which their heads render inside <title> and content="..." attributes. So a crafted link to any blog runs a stranger's script on that blog's domain. Not urgent: blog pages set no cookies and the dashboard session cookie is host-only on blot.im, so it reaches no account; it amounts to phishing or defacement on someone else's domain. Fix: escape the tag (or don't fall back to the raw param) in listingViews.js, switch the partials to {{tag}}, and add a spec for a tag URL containing markup.

Dropbox: a file Dropbox refuses to download (restricted or flagged, a 409 "Conflict") is dropped with no placeholder and no log line: app/clients/dropbox/sync/index.js returns early in the download catch. Write a placeholder and log it, and record it with the file flags once they exist. The check (statusCode === 409 && statusMessage === "Conflict") may also match other 409s; match the Dropbox error tag instead. Test file: tests/files/will_flag_restricted_content.png.

Image jobs run about 3.5x slower in production than on a recent Mac (12 Nov 2026, animated GIF thumbnails)

• About 2.3x is the CPU (Graviton2, Neoverse-N1) and about 1.5x Alpine's libvips build, resizing above all. The allocator (mimalloc, jemalloc) is under 10%. A newer instance type would speed up every image job.

Prove memory of unit fits into cap

Build add font ui

Could we possibly consolidate storage assets and git s3?

• Any opportunities for shared primitives?
• These were written in parallel so there may be overlap

Can we build templates in s3 with their own git repos? Forks as patches? Update application possible?

Where possible sync clients create progress bars

• e.g. when multiple downloads and builds start stacking!

Noticed while fixing the flaky specs, 7 Nov 2026

• Blog.set(id, { cacheID: Date.now() }) is repeated in app/models/template/util/cacheID.js, bumpCacheID.js, rebuildEntriesOnNewHosts.js, app/sync/rebuild.js, app/templates/index.js (twice) and scripts. Since #2666 Blog.set bumps to max(now, previous + 1) whenever a cacheID is passed, so the value is only a request: give them one bump helper.
• Error forking template Error: No template: SITE:blog and Error updating template CDN manifest Error: No template: SITE:blog are logged in the node CI logs of most runs, master included, without failing a spec (and Error in renderView for style.css: Error: No template: blog_<id>:blog in run 42395253798). Find which specs create blogs before the SITE templates are built, or whether a stale read is behind it.
• Other specs may set a claim, lock or expiring key with a TTL of a few hundred ms and assert it is still held, as scheduleSubscriptionEmail.js (#2645) and scheduleWelcomeEmail.js (#2664) did. grep -rnE "PX: [0-9]{2,3}\b" app --include=*.js under tests/ finds them; give the key a long TTL and shorten it with pExpire when the spec wants it gone.
• Many app/clients/git/store/tests/*.js specs create a cache (createCache) in a temp dir and never close it before removing the dir (check, commit, faults, history, index, materialize, reader, remove, s3, among others). The cache writes procs/ files on a timer and walks its directory in the background, so removing the dir can fail with ENOTEMPTY, as scratch.js did (#2644), and the heartbeat timers leak. await cache.close() before fx.remove(dir).
• app/storage/folder/cache.js: a scan whose listing predates removeBlog puts the deleted blog's entries back in the index until the next scan. Same shape as the entry evicted during a scan that #2646 fixed; drop names removed while the scan ran.

Noticed in the first app.log after #2628, 7 Nov 2026

• dropbox:retry (app/clients/dropbox/util/retry.js) logs every error its filter sees, including the ones isRetryable turns down (401 needing a reconnect, 409 folder missing, 507). So each deploy, whose startup syncs touch every Dropbox blog, logs one such line per blog that needs reconnecting. Log only errors it will retry, or log once when the last attempt fails.
• Google Drive: a few Google Drive Client Error: Setup timed out per deploy, at checkWeCanContinue < finishSetup. Find which setups time out after a restart and whether they recover on their own.
• Two deprecation warnings at startup: Node's url.parse() (DEP0169) and moment's "value provided is not in a recognized RFC2822 or ISO format". Find the callers and move them to the WHATWG URL API and to an explicit moment format.
• opensock: Could not establish a connection to 169.254.169.254:80 at unit start: something, probably in the sandbox, tries the instance metadata address and the firewall refuses it. Find what makes the call and stop it, rather than leave a refused connection on every start.
• With openresty's error.log at notice (#2628), auto-ssl's "domain not allowed - using fallback" notices are most of what is left. If they are not worth reading, log them at info.

Noticed during the async/await migration (#2559), 6 Nov 2026

• A unit can briefly lose Redis just after a deploy: on 7 Nov green came up at 05:33:59, its event loop sat near 100% from 05:34:21 (cold caches, one blog's template loading), the Redis client got no reply for 6 s (SocketTimeoutError ... 6000ms), went offline, and 5 blog requests got "Redis unavailable" 503s at 05:34:48. The 03:32 and 04:36 deploys that night had none. A second burst, with no deploy: 7 "Redis unavailable" 503s on a warm blue (up 30 minutes) at 07:49:51, 10 s after Redis began a background save (fork 0.14 s, save 30 s); the 05:34 burst did not overlap a save, and saves run every 5 minutes without one, so it is a data point, not a cause. A third, which points at the cause: at 09:11 a crawler sent about 1,350 requests in one minute to one blog, green's event loop sat at 100%, requests to that blog took 5-15 s (three 504s), and Redis calls on the same unit hit the same 6 s socket timeout (Rate limit and lock-probe). So the Redis client times out because the unit's event loop is saturated, not because Redis stalls. Look at per-blog rate limiting or shedding for crawler bursts, and at what makes that blog's pages slow to render. Find what blocks the loop on a cold unit (a slow render probe on the first minute after start), and consider keeping a new unit out of rotation until its loop settles, or raising the socket timeout during warm-up.
• The dashboard's folder view answers 400 for a path that doesn't exist (e.g. /sites/<handle>/folder/Does%20Not%20Exist): on ENOENT app/dashboard/site/folder/index.js renders dashboard/folder, a view that hasn't existed since #795, so the render fails and the error handler's default 400 goes out. Render dashboard/folder/directory with an empty listing (or 404) instead.
• app/sync/fix/tag-ghosts.js passes a callback to multi.exec(). In node-redis 6 that argument is execAsPipeline, so the tag repairs go out as a pipeline, not one MULTI/EXEC transaction. Call exec() with no argument if the repairs should be atomic.
• Two importer helpers, app/dashboard/site/import/helper/each_el.js and process.js, have no callers. Delete them.
• A logged-out dashboard tab polls /sites/<handle>/status forever: one site's status stream drew more than 8,000 401s today, about 1,500 an hour. The client's ReconnectingEventSource reconnects on any close. Stop reconnecting (or reload to the log-in page) when the server answers 401.
• POST /stripe-webhook answers 503 when a Stripe customer has no subscription ("Failed to retrieve Stripe subscription ... does not have a subscription"). Stripe retries 5xx for days, so a permanent condition becomes a retry storm. Answer 2xx (or 4xx) for permanent errors and keep 5xx for transient ones.
• Entry.get, Entries.getRange/getRecent/getAll call back with no error argument (callback(entries)), so errors are silently dropped and every caller needs a hand-written adapter (blog/lib/models.js, entry/search.js, ...). The migration gives them promise APIs (#2559); until then, treat an empty result there as "maybe an error".
• The PR benchmark's render peak RSS is bimodal (about 470 MB or about 530 MB on the same code), so it flags "regressed" at random. Pin the runner, raise the iteration count, or compare against the same runner's previous run.
• POST /sites/<handle>/template/new (app/dashboard/site/template/index.js) calls Template.create(blogID, name, callback) without the metadata argument, so it has always failed create's argument check (a 500). The "new template" page only posts to /new/upload, so the route looks dead: delete it, or pass {} if something still uses it.
• The Google Drive client retries a sync every few seconds for a blog that can't sync ("Sync init failed ... Cannot sync blog", 7 times in 30 s from both units for one blog on 6 Nov). Back off, or stop, when the blog is disabled or gone.
• GitHub reports 20 open Dependabot alerts on master, 2 critical. Nothing in CI checks npm advisories (see the stale dependencies note further down).

Follow-ups from the ESLint and Prettier setup (#2508)

• app/clients/icloud/macserver/httpClient/resync.js: the return inside finally (when the debounce entry is gone) swallows a failed resync, so the caller sees success. Move the cleanup into an if (entry) and let the error through; needs a macserver deploy.
• app/clients/google-drive/util/download.js and scripts/tests/util/site.js build a new Promise(async ...): an exception thrown outside their own try/catch is lost and the promise never settles. Rewrite both as plain async functions (part of the callbacks to async/await migration).
• app/dashboard/site/import/helper/replace_embeds.js threw a ReferenceError (no cheerio or parse in scope) on every call until #2508, so nothing can have been using it successfully: check whether the importer calls it at all, and delete it if not.
• app/documentation/tools/finder/index.js threw on a code.file element without a type class (missing determine_icon require). Now fixed, so the docs may render icons that were never exercised; check the documentation pages once.

Host cycle follow-ups, from the live cycle onto a 30 GB root on 6 Nov 2026 (#2552)

• Check the daily email of 7 Nov: the first offsite (B2) sync from the new host runs at 02:10 UTC. Overdue or failed means the blot-b2-storage profile or /etc/blot/storage-offsite-bucket did not reach it at app.
• The final data-volume snapshot (tagged purpose=final-before-removal) is the last cost left from the volume: move it to the archive tier around 20 Nov, or delete it once S3 plus B2 have run for a while. The maintainer decides.
• launch.sh connects with StrictHostKeyChecking=no and keeps no key: have it read the ED25519 fingerprint from get-console-output, check it, and write a known_hosts line for <new> (the README says how to do it by hand).
• Time each step of switch inside the freeze. Writes were frozen 110 s live against 41 s in the drill, with no failed requests; probably ssh round trips through ProxyJump from the maintainer's Mac, unmeasured. Then decide whether to run the in-freeze steps from the new host instead.
• cycle.sh deploys the old host's image, so it could set SKIP_BRANCH_CHECK=true for units.sh itself instead of failing from a worktree.
• The Deploy workflow's commit input fails on a short hash ("could not resolve commit") although its description says short codes work: fetch enough history, or say full hash only.

Serve blog folder files on the CDN straight from S3, as assets are (left over from #2258, closed once the B2 restore test passed)

• Bunny's allowlist edge rule may still list /html2canvas.min.js, which #2661 removed (the route and file are gone). Harmless; drop it next time that rule is edited.
• Today cdn.blot.im/folder/v-<version>/<blogID>/<path> reaches the origin, which reads the file through storage/folder. Every blog is on S3 now, so a Bunny edge rule could send /folder/v-* to the bucket instead, like _image_cache and _thumbnails.
• Decide first: dotfiles (node 404s any path segment starting with ., S3 serves all but .git, so maybe Deny .env-style names too); directory URLs (node redirects to the trailing slash and serves index.html, S3 has only a zero-byte marker: check whether the build ever bakes a directory link); a missing object is a 403 from S3, not a 404; Content-Type comes from the object.
• Watch #2258's performance gates (S3 requests, origin hits from the [folder-asset] log line) before and after.
• Keep /folder/v-pending/* on the app: it finds the file ignoring case and sets its own short cache times (see app/cdn/index.js), and its 404 must not become S3's 403.

Why does helper/ have four ID generators? can we consolidate under one, using a backfill?

Fix this: Callbacks and promises mixed: about 199 files mix them, and async@2 is required in 72. This causes double callbacks and dropped errors.

Update these: Stale dependencies:

• stripe@6.36.0, a 2019 client, on your payments path.
• highlight.js@10 (end-of-life) and jasmine@3.3.1.
• faker@4.1.0 and colors@1.3.2. Their author sabotaged both projects in 2022, and you still carry them.
• moment, debug@3, async@2, node-fetch@2.
• You have a skill that checks Node, nginx and the OS for advisories, but nothing checks npm packages. The checking points at the wrong place.

Fix this: 879 console.log calls in non-test app code, and no structured logging. Several of your skills exist mainly to grep and rebuild meaning from free-text logs ([STATS], [SLOW-FILL], [folder-asset]). Structured logs plus a metrics backend would make most of them unnecessary.

Fix this: Weak input validation. The blog schema only type-checks client as string. In app/dashboard/site/client.js, POST /switch disconnects the current client, then saves req.body.client without checking it names a real client. Later, clients[req.blog.client].disconnect hits undefined. It’s minor, but it’s the kind of thing a reviewer finds in five minutes and generalises from.

Unify this: Duplicated libraries: node-fetch alongside native fetch, moment throughout, and some scripts require request, which isn’t in package.json.

Unify this: Scattered config: about 101 environment variables, and 14 files read process.env directly.

Fix this: Dev differs from prod: the test Redis is redis:alpine, unpinned (scripts/tests/invoke.sh:30), while CI and production run 6.2. Markdown goes through pandoc in production and through marked in development. The sandbox runs jailed in production and in-process in development.

Remove customer info from TODO (and gmail links)

Research then implement this if sound: Make the version the only invalidation. Put cacheID in the proxy cache key, and drop the purge RPCs, the pending-purge queue, the per-file purges during sync, and most of the LRUs. Old entries age out by eviction. This is cheap and removes a whole failure class. Full pre-rendering is the logical end point, but don’t go all the way: search, arbitrary query strings and very large catalogs make it a lazy-render-into-versioned-store design, not an SSG.

• I wonder if we could store the cacheID against the host and then as the ssl cert was read, we could seed this info automatically?

Switch from redis to relational db?

• Per-blog SQLite on object storage

Implement: 5. Builds in isolated workers off a job queue.

• multiple builds in parallel

We should follow sane soc 2 practices

---

Render and client-side cache follow-ups (#2390, #2401, #2420, #2426, #2434)

• Post-deploy check of #2434 (deployed 3 Nov 2026, 11:21 UTC): node-a's [STATS] show redis_cache_maxEntries=10000 and redis_cache_<class>_evictions for the capped classes, and node-a's render p95 (metrics:render-time:p95:<unit>/node-a, read as scripts/render-time-chart does) matches the previous 24 hours, about 13 ms
• From 4 Nov 2026, a day after #2434's deploy (3 Nov, 11:21 UTC): in node-a's [STATS] in both units, check the client-side cache's redis_cache_evictionCount against the sum of the redis_cache_<class>_evictions. A cache-wide count above that sum means blog keys no longer fit: raise maxEntries in app/models/redis.js. Also check redis_cache_entryCount against the 10000 cap, and each capped class's evictions against its CLASS_CAPS share (app/models/redis-cache.js)
• In the same lines, check the redis_cache_session_* and redis_cache_user_* hit rates. Every process starts with 63 user misses, one read of a batch of users at boot: find what reads them, and whether it needs to
• From 10 Nov 2026, a week after #2426: no reports of early logouts, and sess:* TTLs stay at or under 30 days (disableTouch plus a fixed ttl, app/dashboard/util/session.js)
• Then tune the render cache caps (under Memory below)
• Run the mega benchmark (scripts/benchmarks/mega) twice before merging any render or client-side cache change

---

Memory: cut each unit's total ceiling (the unit container plus its sandbox cgroups, about 4.8 GiB today now the airlock is gone: 3300m unit, 512M image jobs and proxy jail, 1G browser) to about 3 GiB, then aim for about 2 GiB of real use by shrinking Node itself.

• Done: the airlock is gone (#2257), the sampler splits node-a, node-b, other node and git (#2413), and git commands are bounded per process (#2415).

Unit and sandbox limits

• After #2415 is deployed, collect a few days of unit-mem.log including a deploy and a busy period, then read unit-mem-summary.sh: node-a, node-b, other node and git maxima, the sandbox cgroups' peaks and oom_kill, and the host's lowest mem_available.
• Then lower UNIT memory and the node-a/node-b heaps in scripts/deploy/constants.js to about 2000 MiB, with the node-a and node-b heaps around 1100-1200 MiB (keep V8's heap limit below the cgroup's, so a runaway heap dies with a report, not to the OOM killer), and the sandbox cgroups in config/host/host/setup.sh to about 384 MiB (image jobs and proxy jail) and 700 MiB (browser; real pages peaked at 417 MiB, the heaviest measured needed up to 880 MiB). Accept a few heavy pages failing in the browser jail; sandbox failures email the admin.
• One-off scripts run with docker exec count against the unit: run anything heavy through scripts/probe (its own container and limit) instead.
• Is 512 MiB enough for the image jobs, given four share it? The proxy jail (128 MiB) and the fetch-image job (256 MiB, one at a time, #2424) are in the same subtree, so the image jobs take the rest. cgroup peak against limit and oom_kill above 0 in blot-sandbox-blue/green say; grep "broker: job=" in the unit logs for what failed.
• The mid-Nov reservation check (host items below) reads the same unit-mem-summary.sh output: do the two together.
• Does the deploy (scripts/deploy/units.sh) check the host has enough free memory and disk before it starts a new unit?

Node heap and caches

• Tune the render LRU caches, which now live in app/models (#2401, #2420), from the render_cache_* counters and [SLOW-FILL] lines in the [STATS] logs (#2233). Include the mustache template cache. Keep the config identical on node-a in blot-blue and blot-green, sized so it fits node-a's heap after the cut (1100-1200 MiB, of which the client-side cache takes up to ~25 MB, 80 MB ceiling, see app/models/redis.js; before the units the worst case was ~620MB on 27 Oct). node-b in each unit runs with small caches via BLOT_CACHE_SCALE, so it is out of this tuning. Pull the counters from node-a in both units (ask before each prod command) and per cache check hit rate, evictions, fill ms (fillMs/fills, fillMaxMs), uncachedFills and which blogs the [SLOW-FILL] lines name. Then investigate an admission rule: only store a fill in the cache if it took longer than a threshold (5ms, probably per cache; fullView and getAllCached are far slower than the rest, which were mostly ~0ms). Cheap fills would stop evicting the expensive ones. Things to settle: the threshold must beat the cost of a hit (every hit does a cloneDeep); cost times frequency may matter more than cost alone; fill time includes event-loop delay, so smooth it or add hysteresis; keep coalescing concurrent fills (store then delete, or an own in-flight map as getAllCached has); a nashp-sized catalog is slow but huge and may not fit maxSize. A two-tier cache or GDSF-style cost-aware eviction are heavier alternatives. Once the capacity counters (missGhost/missCold, evictIdle, hitAge, uncachedBy, cloneMs) have run a day or two on prod, size each cache from them: a high missGhost share means more room would help, a mostly-cold one means it would not.
• Heap creep: before the units, green's baseline heap crept up ~40-80MB/day (lowest heapUsed in [STATS]) and never reset until a V8 OOM crash (13+14 Oct 2026). Check whether node-a's and node-b's lowest heapUsed still climbs between deploys; if it does, find whether it's a leak or an unbounded cache by comparing heap snapshots taken hours apart. Spikes from heavy post builds (e.g. blog_825ed2a 'All paths', ~250MB) only crash it once the baseline has crept up
• Later: shrink Node's own memory (the render LRU caches, the heap) toward about 2 GiB per unit in real use; on 2 Nov the unit's own unreclaimable peak was about 1.4-1.6 GiB, with the biggest Node process about 1.15 GiB.

Diagnostics

• Per-request resource monitoring, e.g. heap size before/after, to find which requests and builds cause the spikes
• CloudWatch memory metrics on the instance and from the app, or are unit-mem-log.sh and the existing alerts enough?

---

when we next cycle hosts, should we cycle the ssh key pair too?

• 3 Nov 2026, after 03:05 UTC (#2261): the first daily Redis backup from the new host is in S3 and B2, and no stale-offsite email was sent
• 8 Nov 2026, after 03:35 UTC (#2261): the first weekly/ Redis dump from the new host is in S3 (aws s3 ls s3://blot-redis-backups/weekly/). On 3 Nov there was no weekly/ prefix at all: no weekly dump had reached S3 since #2055 set them up on 7 Oct (the lifecycle rules only expire weeklies after 30 days, so they weren't swept), leaving only 7 days of dailies to restore a deleted blog from. If it is missing, read ~/backup.log on the host and fix it before the next Sunday
• Next wildcard certificate renewal (#2261): it works with the new blot key; its Route 53 access wasn't tested
• For a week or two after the 2 Nov move (#2261): watch [LOCK] slow heartbeat lines and the Redis slowlog. If they appear, pin Redis to one core and limit the units to the rest
• Cycle the production host (#2377) onto the 30 GB root disk, since a root volume can't shrink in place. Rehearse config/host/cycle.sh with --drill first: it has no data phase now, and switch freezes the app itself
• iCloud sync is flaky: on 2 Nov, after the host move, a new folder in icloudtest never reached the macserver within the test's wait, though the macserver's calls to the new host succeeded. Find out where iCloud changes stall (iCloud itself, the macserver's watcher, or the resync path) and make sync dependable
• Mid-Nov 2026 (#2261, #2107): buy the m6gd.xlarge reservation, since the host runs on-demand until then. First run unit-mem-summary.sh on the host over a stretch that includes a deploy, to confirm the 3300m unit limits and 6gb Redis maxmemory fit the size Git store (#2259 follow-up): raise the push limit (the issue has the numbers) Git store (#2259 follow-up): a warm push still makes one avoidable round trip, the manifest read that the advertisement already did Git store (#2259 follow-up): show push progress through the proxy Git store (#2259 follow-up): set socketTimeout on the store's S3 client Git sync (#2259 PR 7): when the last applied commit is gone from the git store (a force push, then a rewrite), a sync compares from the empty tree: it writes every file but removes none, so files the rewrite deleted stay in the blog folder until a resync Git (#2259 PR 6 follow-ups): a branch other than master that an earlier push got in (the bypass) can't be deleted by its owner, since every non-master branch ref is refused, deletions included Git sync (#2259 PR 4): a sync interrupted part way leaves already-applied paths ahead of the recorded commit, so a later push that restores such a file to the recorded commit's contents shows no diff and the entry stays stale until a resync. Since the materializer (PR 7) the file in the folder stays stale too, because nothing resets the folder. Needs a crash or Redis outage, no retry, and that exact revert

Speed up ci / simplify proxy and e2e testing slightly

Deploy macserver

---

Set up Redis redundancy using a primary on the main server, a replica on the Frankfurt server, and three Redis Sentinel instances (main, Frankfurt, Mac) for automatic failover with quorum 2. Configure Node.js to discover the current primary via Sentinel. When the original main server recovers, it should rejoin as a replica; restore it as primary through a controlled manual failback using SENTINEL FAILOVER.

Proxy follow ups:

• Can we modify the proxy disk_cache to key requests off blogID in some way? SO we could also cache preview subdomains and purge them efficiently?
• Does proxy/ send and recieve etags nicely? Can we precompute these for compressed resonses?
• Can we exploring filling the disk_cache from node? use background task to do this for other pages by cache ID? When everything is in memory in the node renderer, why not?
• Are the nvme disk settings on the host tuned for max performance?

Analyze the new log line paramater for accept encoding to determine if we should precompress gzip or not:

• Check the access log for the Accept-Encoding of feed requests (/feed.rss etc.): if most are gzip-only, store gzip -9 copies too (gzip_static), since those hits still compress on the fly,

Redis host follow-ups (from #2041, new host since 8 Oct):

• Lock heartbeat: no overlapping EXTENDs
• Remaining dropped write errors: await set() in resetToDrive; Dropbox disconnect.js and _assign.js dropFromMenu check the wrong error; write order in Blog.create; the list-ghosts restore path; error paths for background resave and rebuild; a callback for the messenger's setStatus; make Entry.set atomic
• TLS handshakes were about a quarter of billed egress on 21 Oct (#2107): ECDSA certificates shrink each by about a third, and few TLS connections resume today, so look at session reuse and keep-alive (about 1.7 requests per connection). No ssl_session_tickets without a shared key (#2271)
• restore.sh --offsite: have the host download from B2 with a short-lived presigned URL (aws s3 presign on the maintainer's machine, passed over stdin), so a restore doesn't depend on the maintainer's upload speed and the read key still stays off servers. Not tested yet
• cycle.sh --drill hosts have no AWS credentials (no instance profile), so restoring an S3 backup onto one means copying keys onto it or using --offsite, which streams the dump through the maintainer's machine: on 3 Nov a 2 GB dump took about 30 minutes at 1.2 MB/s. Give drill hosts a read-only instance profile scoped to s3://blot-redis-backups (or reuse the presigned-URL idea above for S3 too), so the host fetches it directly in about a minute
• Upgrade the Redis server from 6.2 to 8.4+ for native compare-and-set (SET ... IFEQ, DELEX), then drop Lua: helper/lock extend/release become plain commands; helper/alarm claim/renew/complete need per-id claim keys to go native; also the legacy scripts in sync/lock.js, models/user/set.js, helper/flushCachePending.js, clients/icloud/database.js

Proxy container follow-ups (cut over 8 Oct 2026, #1941)

• Check webhooks. SSE through the container, and that a known /blog_* asset on cdn. has Cache-Control 1y and no Blot-Upstream
• After the next 03:46 logrotate: access.log/error.log still written after copytruncate, fail2ban counts still moving
• After the first wildcard renewal by the app (scheduler log "Wildcard SSL"): the proxy serves the new cert fingerprint from Redis within ~10 seconds, with no reload ("wildcard cert:" lines in the proxy error log mean it fell back to the file)
• Custom-domain certs renew inside the container: none within ~20 days of expiry apart from domains that no longer point at Blot; after a daily renewal run, error.log should show no "certificate for ... expired at" warnings for domains still pointing at Blot (the proxy now refuses to serve an expired stored cert; no host cron)
• Reboot the host once in a maintenance window: mount-instance-store mounts without reformatting, docker starts after it, the proxy comes back with cache/log mounts, :80/:443 answer
• http://www.blot.im/ returns 404: confirm intended
• app/blog/static has no copy on the host, so those cdn. files fall through to Node: export them from the Node image on deploy, or accept
• Get proxy horizontally scalable and multi-region
• Can we add modern compression to proxy? (brotli is in; zstd is not)
• Can the proxy remember bad urls and avoid repeatedly sending them to upstream?
• Replace host logrotate (config/host/logrotate/nginx-{access,error}: rotate 0 + copytruncate, i.e. truncate at 150M) with an ngx.timer in the proxy that truncates access.log/error.log over a size cap; check workers (ec2-user) can write the log files. fail2ban stays on the host
• Tune proxy next upstream status code conditions to preserve zero-downtime-deployments but prevent expensive requests (timeout?)
• Add support for IPv6 and tell Raul
• Verify that nginx config serves as many CDN static files as possible
• Re-set up html minification using html-terser?
• Get blot proxy communicating with node process via unix socket

---

Finish the host consolidation

• should we use unix sockets for communication with redis?

How to reduce likelihood of downtime?

• multi-region (dns level failover from one region?) with redis replica in another region
• how to prevent dns issues?
• are we well insulated from bunny failures (dns failover here too?)
• are we well insulated from cloudflare failures (they do our DNS records or is it route53?)
• are we as well insulated from most likely aws issues?

---

Harden app against S3 being unavailable (generated assets and blog folders)

Implement folder storage limits (include generated files in this amount?). #2258's Redis index keeps the per-blog byte totals to enforce them from.

Bump katex to >= 0.18.2 (low, prototype pollution Dependabot alert).

Sandbox follow-ups (#2257):

• Hardening: a seccomp filter for OpenResty, like Node's (proxy/unit/supervisor.sh, proxy/unit/sandbox/node-filter.js): its workers parse hostile requests and it is the one unit process left unfiltered.
• Hardening: a CI job that runs the real jail (nsjail, pasta, the seccomp profile and a cgroup) against the unit image. Today the jail is only exercised by the deploy's verify-container checks on the host.
• Retire the proxy jail (tinyproxy, proxy/unit/sandbox/proxy.js, egress-proxy.nft, BLOT_EGRESS_PROXY_URL, the proxy check in app/sandbox/alerts.js and verify-container's "sandbox proxy", sandboxProxyPort): the fetch and fetch-image jobs replaced every fetch through it. It frees its 128 MiB cgroup, which is when the image jobs' share of the unit's sandbox subtree grows.
• Count successful sandbox jobs, not only failures: the broker logs only failed jobs, so how much each job is used shows only in the sampler's cgroup peaks.
• Before dashboard screenshots of customer sites and template previews: size the browser cgroup and queue for that load (one Chromium per unit at a time today; heavy pages need up to about twice the median), from the sampler's blot-sandbox--browser lines.
• CJK fonts for the browser jail, so sites in those languages don't render as boxes. They are large, so weigh them against the unit image's size.

---

Future idea: an external link checker for the file flags (#2673): a scheduled job which checks the links in posts to other sites and flags the file BROKEN_EXTERNAL_LINK (with each URL, its status and when it was checked). It means fetching links on other sites from our hosts (egress, IP reputation, internal addresses), so it would go through the sandbox's fetch job, at a low rate per host, with results cached across blogs, flagging only after repeated failures.

Add a new template which just looks like finder on the index page

How could we generate the directory tree structure used in the hypertext template sidebar on the Server in the template engine

Create infrastructure to curate an explore page (keep it hidden for now)

Create skills to moderate /questions

Suppress new site admin notification for no subscription change (i.e. user deletes then adds back in a single site)

Add an arbritrary input: import this site which just emails me

Add one time login cancellation links to renewal emails

Add lifetime subscription to the billing interval page

Add multi-currency subscription signup with fixed, rounded market prices for EUR, GBP, JPY, CAD, AUD, and a catalog that can expand to more currencies. Keep the catalog as the source of truth and add idempotent scripts to create/reconcile Stripe Prices and PayPal plans, recording provider IDs without changing plans used by existing subscribers. Update signup to suggest a currency from geo-IP while letting customers choose, then select and verify the matching provider price server-side.

Implement liveline for stats streaming

• https://benji.org/liveline

Add a tool which randomly samples, production markdown posts, and runs version of build and version B of build on it

• Goal would be to identify regression in a new version of the build process

Add transfer site flow to blot from one user account to another, somewhere in settings?

• for designers and develeopers to transfer a site to their client after building it

Onboarding:

• The default title of the person's first site should be just the bit before the @ in their email?
• Use gravatar to fetch their name and use this for the title of the first site. Use gravatar to fetch their avatar and upload this as profile_image

Improve daily update email:

• Add daily aws cost to email update
  • Set up AWS cost anomaly detection

Open source as content marketing

• Close my GitHub account
• Release pane and other open source projects that I can spin out of blot

Add rebuild button to each file on dashboard

• Add json viewer to files on dashboard

benchmarks-render.yml is manual only because benchmarks-corpus.yml fails, leaving no corpus: fix the corpus build (probably needed since #2111 dropped data/static from the corpus), then give the render benchmark a daily schedule

Bunny: decide Volume vs Standard tier (more PoPs, faster first byte far from Volume PoPs) once we've run the numbers on cost and latency; also why cache HIT is low (Statistics: status codes, top uncached URLs)

Add a list of billing history / invoices to subscription page

Create skills which delegate cheap subagents to scan logs for issues:

• proxy logs (focussed on error)
• node container logs

Upgrade the Stripe integration (low priority): the stripe npm package is 6.36.0 (2019) and the account's default API version is 2011-11-17, except app/models/user/syncPaymentMethods.js, which pins 2020-08-27. Move to a current stripe-node and pin one explicit API version on every client, after checking the webhook handler (app/dashboard/webhooks/stripe_webhook.js), subscription create/cancel/pay and card update against the API changelog. Upgrade the account default in the Dashboard only once the code no longer relies on it.

Do we eventually prune and rotate the data gathered in the various statistics cron jobs? Eventually we need to purge these we don’t want the database growing infinitely

archives and all_entries in the template engine should be bounded to a maximum of 10,000 posts

• separately we should investigate other parts of the code which perform unbounded work

Maybe horizontal scaling involves redis replicas in other aws regions?

Status page:

• Is it just measuring the LRU caches? Make it verify a cold request that skips the caches and goes all the way to Redis (maybe through a synthesised route), to get a real sense of system performance
• Add client status (check api)

Add email warnings if no sync from client for more than x hours

---

Internationalize the dashboard and brochure site with a standard catalog-based i18n library (e.g. i18next), locale-aware formatting, and a translation workflow. For user templates, build a small compile step that turns localized source/catalogs into ordinary Mustache views, keeping templates portable and easy to extend with languages. Start with major European languages plus Chinese and Japanese; decide Simplified/Traditional Chinese scope and phase brochure content by priority.

Overdue and cancelled (i.e. non deleted) sites under X GB should have their data exported, archived in s3 (download link in email) for up to a year after closure.

• Make node scripts/user/delete-users-to-remove.js -fast happen automatically

Add macserver deploy to github actions

• Consider making the deploy via github much safer – how?
  • remove ssh keys from deployment. maybe set up a minimal deployment agents on the servers? maybe autodeploy images with a given tag?
  • how do we add 2fa to deployment flow, require email confirmation somehow? onetimelink? assume github could
  • if github is owned are we owned? ideally not

hypertext template on indexes:

• title needs to link to permalink page
• anchor links in title need to link to anchor on permalink pages
• improve aesthetics and spacing of pagination links

Could we improve the performance of posts/pages with lots of images by adding loading=lazy in the output HTML?

Make sync logs more useful, surface recent changes on dashboard. Group multiple files together if changed at same time, etc. make it useful!

• Add error / getHealth issues to sync log (folder.status) for Dropbox/other client Issues

Excercise the backlinks with big entries / many backlinks in the benchmarks somewhere

How do we think about a shared asset library across templates?

• e.g. shared favicons
• can we unwind some of the complete delete favion on template delete, etc. code?

Decide whether or not to revert the preset system – should it only be used for the templates index for explaining the customization features?

Test script for point-in-time restoration of specific blog folders at specific time (e.g. 2 days ago), from the B2 copy of the bucket (restore-folder.sh restored from EBS snapshots and went with the folder cleanup)

Under templates index

• Clicking on file in folder pulls up post on preview site ?

Clarify on cancel / delete / overdue pages for accounts with grandfathered subscription prices that if they re-join Blot they will lose their grandfathered rate.

Sync sweep digest (app/clients/util/syncReport.js) follow-ups

• Hourly Fix() (Dropbox, iCloud, Google Drive) runs outside the folder lock, so menu-ghosts' whole-array menu write can clobber a concurrent sync's menu change - fix in app/sync/fix/menu-ghosts for all clients (raised on PR #2001)
• iCloud: stop watching blogs that are disabled or gone (the macserver keeps uploading, gets 500 Cannot sync blog from sync/index.js and requests resyncs); have /upload answer 4xx so the macserver drops them
• iCloud initial transfer: the verify pass now reruns twice with a pause, but still gives up with TRANSFER_INCOMPLETE if every attempt fails; make initialTransfer resumable (a stalled ls during the upload burst caused TRANSFER_INCOMPLETE). Check the macserver's new ls took/Diagnostic snapshot/still running after kill log lines for the root cause of the stalls
• Don't report a stuck lock for a blog still in its initial setup/transfer (Dropbox transfer_pending, Drive preparing, iCloud transferringToiCloud) - a big first transfer can legitimately hold the lock >1h
• Once-per-occurrence suppression treats "not seen this run" as cleared, including blogs that just weren't eligible that hour, so an ongoing error can be re-emailed after a quiet hour - remember issues until a successful walk/Fix actually clears them

Surface template choices on the templates subpages: switch from png to iframe after hovering a selection

Perform caching and header audit of site and templates

• Check logs and headers via curl
• Identify optimizations

Make sure next bill amount is in all billing emails to customers

Explain on the about page the websites started out as files in a folder. But they had to be HTML. Blot makes other file types possible too.

Get node container horizontally scalable and multi-region (#1932)

Add clear bill change info to new blog flow

Rebuild all sites to bake in folder CDN urls

Why is Ross' template so slow to install? It takes a few seconds. Let's speed it up.

Improve about section

• Restructure /about into current principles and clearly dated working notes.
• Rewrite the main page to surface Blot’s purpose, independence, longevity, portability, and open source code.
• Update History, infrastructure, development tools, refunds, backups, and continuity claims.
• Date historical material such as the CDN investigation, old mistakes, and threat assessments.
• Remove outdated self-undermining statements, including dissatisfaction with the templates.
• Fix incomplete passages, spelling errors, and inconsistent use of “I” and “we.”

Fix performance bugs on various sites:

• anchor and re-enable site: slow from entry size, not count; every cacheID bump reads full entries for posts (page size 100) and getAllCached just to print titles/URLs. Modelled as the "heavy" site in the mega benchmark (scripts/benchmarks/mega, findings in data/mega-shape/findings.md)
• nashp (improve speed): every page pulls the whole catalog unprojected plus all tags, and previews of its many templates skip the catalog caches. Modelled as the "catalog" site in the mega benchmark
• jaysjay (especially archives)
  • entry pages take ~3.8s on every render, warm or cold: ~1000 Redis commands / 18MB per render while loading entries (render-probe, 2026-10-06)
• heydingus /search 0.5-0.8s warm (search_results reads ~10MB per query); its template emits …/null links that crawlers 404 on
• blot.im /questions pages 250-750ms, mostly I/O (elu 0.2-0.4), incl. 302s from /questions/:id/edit
• using additional req trace logging and more possible LRU caching
• do we need a seperate incrementally updated index of skinny entry properties, e.g. title url etc for rendering archives / allEntries?

Fix benchmark build-corpus action issue

---

Improve Google Drive sync

• Fix Google Drive sync outage caused by Google account suspesion
  • Future fix to Google Drive client needs to never use the service account to create files (ask the user to handle the initial transfer)
  • Revert 1958
  • Create simple config option to disable a given client when something like the Google Drive issue happens. Make it clear on the dashboard and docs that it's not working.
  • Add google drive kill switch (when platform goes down, etc.)
• Fix bug with Google Drive client and Stephen's large blog
  • It also seems like server crashes when a massive google drive folder is synced, with a directory containing 6772 items
• When you re-connect to Google Drive, you have to re-create a folder and perform a full transfer. It would be great if we could re-use the existing folder... Maybe an option during the connect flow? Would be nice to avoid requiring a full transfer
• Fix bug with shared drive folders for Chuck
• Fix bug with Google Docs when disconnecting a site from Google Drive on Blot then reconnecting the site to Google Drive. We need to work out how to handle this (Google Docs are not real files)
• Improve sync efficieny with option to target a specific folder or file first (Track ids in database) and tell Chuck maybe use touch metadata method to force syncs periodically for hot files?
  • Make it possible to short-circuit the full folder traversal and immediately begin at a specific file/subfolder
• Improve html output for google docs and tell Chuck Issue:

  
  

  should be

• Fix possible bug where it seems like sometimes the first sentence of Google Docs become titles unexpectedly – possibly a mis-identified title tag?
• Surface storage quota errors as QUOTA_EXCEEDED health for Google Drive (isQuotaError exists, nothing sets the code)
• Retry 429s on Google Drive writes (files.create/update): reads already retry with backoff (serviceAccount/retryConfig.js), but write bodies are streams gaxios can't replay
• Add a way for sync/build to accept a ctime as an option. We can't set this at the operating system level but google drive's API does return a createdTime value and we could use this as the true file's created date, rather than the date the file is made on the server
• Google Drive write should use the best method for large file uploading
• Add proper testing for this
• Make Google Drive incremental sync efficient
  • Store hash of their api responses
  • Don’t touch disk
  • Support huge folders with huge directories
  • Start with recently changed files and dirs first

Improve Dropbox sync

• Batch dependent-entry rebuilds across a sync: a large asset delta batch rebuilt the same referring entry after each image (~13s per update), holding the folder lock for over three hours (2026-11-04). The stuck-lock alert should distinguish ongoing progress from a hung sync.
• Review public asset rate limits and nginx-429 bans: loading an image-heavy page triggered repeated 429s and temporary visitor bans (2026-11-03/04); avoid banning normal page loads while retaining abuse protection.
• When you re-connect to Dropbox, you have to re-create a folder and perform a full transfer. It would be great if we could re-use the existing folder... Maybe an option during the connect flow? Would be nice to avoid requiring a full transfer
• Sync verification should instrument the sync/fix function to also report errors where the folder doesn't match the database.
• Bug: ensure that when a process dies, any blogs setting up dropbox client will be re-attempted. A startup resume was tried in #2515 and pulled: resetFromBlot deletes Dropbox files Blot lacks, so resuming a transfer stuck days ago would delete what the user added in Dropbox since (webhooks skip transfer_pending blogs); and onceAtStartup marks the run done on blot-blue while green's setup still holds the lock, so a setup killed by the deploy waits for the next one. Resume only recent transfers (record when they started), from the hourly validation rather than startup
• Adding an additional blog to an existing app folder under constant webhooks fails: moveExistingFiles (routes/setup/createFolder.js) gives up on the root blog's lock after ~5s and setup fails loudly. Retry or wait longer for the other blog's lock (or take all account-linked blogs' locks in a fixed order)
• Fix bug with Client.write which does not propogate sync errors... to demonstrate this, revoke dropbox token then try to enable local editing.
• Bug: make large Dropbox upload sessions (>140 MB) resumable across process restarts by persisting the session id and offset
• Bug: run reset-to-blot when sync fails / errors
• Bug: ensure you can move the folder in full-folder permission mode
• Bug: reconnecting with the same account and same permissions should not create a new folder for full_access=true
• Bug: add a check for folder storage before doing mass upload
• Bug: handle folder full error appropriately
• Add testing for dropbox:
• Modify setup flow:
  • Phase 1: RESET FROM BLOT (NON DESTRUCTIVE, deduplicate on conflict)
  • Phase 2: RESET TO BLOT (TWICE?) then sync/FIX then begin ordinary syncing
  • Add a progress bar to the dashboard during initial client setup. Use the Google Drive and Git client progress bar implementations as reference.
• Make Dropbox sync issue emails more useful with a script to compile the logs for a single site (including macserver)
  • Adjust full_folder to add a big warning to prevent confusion about where the new folder is when transitioning permissions
  • Add a step to authentication flow which says: 'there is already a folder called 'Your blog title' in your Dropbox, would you like to use that?' then overwrite the files inside. This prevents the proliferation of confusing 'Your blog title (1)' when you re-authenticate.
• When you switch from app folder to full folder, there should be a notification indicating that your folder location has changed
• Check content hash for upload possible
• Make dropbox reset FROM abortable
• Make dropbox reset TO abortable
• Surface errors during setup on dashboard
• Surface sync issues with Dropbox client on the homepage of dashboard
• How do we make errors sticky? To make sure users click through to clients page to re-connect to Dropbox, etc...

Improve git sync

• Create tests for large Git repo initialization
• Make git client setup resumable / handle server restart
• Fix bug with git client which let large pushes overwhelm the main server, leading to downtime
• Handle error out of space on blot server during git client setup
• Add sync check
• Add support for LFS and tell Jamie
• Backdate existing commits (perhaps set file ctime using info from git log?) This would sidestep the frustration experienced by Gavin when he imported an existing repository.
• Extend sync lock during long syncs
• Write test to simulate process dying and restarting mid-pull. Does the client recover? What happens if a pull fails midway? Does git recover?
• Write tests for a series of random client operations (pull, add, push, move, reset, delete) that can be seeded and reliably re-run in the case of failure.
• Move to neat promise chain api when 2.0 comes out
• Test that client must keep sending new entries and deleted entries when it recieved an error from entry.build otherwise you get ghost posts...
• send nice error message with instructions for 413 request too large: "git reset HEAD^" etc...
• Thoroughly document:
• Explain that the git client uses the master branch and repos from github will use main in the docs and on the dashboard then tell N
• Tidy up tests and reduce size and runtime
• This client should acquire a lock on the blog's folder before doing stuff to it more generally, especially during initialization etc...
• what happens if customer changes their handle? will pushing break?
  • I want them to be able to keep using their old one until someone else claims it, then I want 401 unauthorized, please reauthenticate with new username... but the remote path will be wrong?
• Rate limit authentication route?

Improve iCloud sync

• The dashboard needs more feedback as the macserver goes through iCloud setup: it would be useful to echo that the invite link is valid, we're waiting for the new folder, then that step can wait up to a minute while so the user understands what's happening? the google drive client has a step based system
• We need a plan for the suspension of the iCloud user as well... is the code resilient to data loss in this scenario?
  • Add icloud kill switch (when platform goes down, etc.)
• iCloud: the hourly integrity check is back; once it has run cleanly for a while, decide whether to re-enable the daily resync (resyncAllConnected) and the startup//started resync (resyncRecentlySynced), still disabled from these commits
  • 0043d91b36621ecd6aef2bcc263f51f209924792
  • 98dc9405c883825917de8e436ec620f60adaf975
• iCloud server should periodically:
  • quit safari (some error flow during auth sometimes opens it)
  • close all system UI prompts (sometimes we get error for maybe accessing a revoked folder?)
• Sync no-progress watchdog: app/sync/lock.js renews a folder lock for as long as the process lives, so any hung sync holds a blog's lock until a container restart (7h+ on 16 Oct, #2147). Track the sync's last progress (file processed / status update) and, after N minutes with none, stop renewing and fail the sync so the lock expires and the next resync or hourly check can run
• iCloud macserver should periodically validate the local blog folders to ensure the client is still connected on blot and clean up any failed initial transfers (i.e. non blog_... folders)
• If we need to resolve further bugs with sync drift:
  • Implement additional parsing of modified paths from monitorer.js (brctl monitor -i) and reconcile these in addition to the fswatch module with the chokidar events for additional redundancy
• Think about adding a more efficient endpoint to sync folder state.
  • use wisdom from rsync?
  • the call to recursive list can take a way long time for large blogs (blog_3ec96xxx) so we need to increase timeout from 10 seconds to maybe multiple minutes...
    • can we get rid of recursive list entirely if we simply call brctl download on the paths flagged by brctl monitor?
  • don't hold sync lock the whole time?
  • fetch file list, compute changes, then lock, then apply?
    • we'll need to update the call sites since the lock will be handled within
• Daily icloud resync should also log how long the longest folder took to resync.
• Add stats on response time to iCloud server health check logging
  • maybe monitor response time average and email notify when it increases dramatically?
• Smooth over setup process, add guidance for 'mode' of share panel (e.g. copy vs. collaborate) and maybe a screenshot
• Add a progress bar to the dashboard during initial client setup. Use the Google Drive and Git client progress bar implementations as reference.
• Make iCloud sync issue emails more useful with a script to compile the logs for a single site (including macserver)
  • Add logging instrumentation of modified file paths to sync verification to aid in analysis of sync drift
• Enforce a configurable timeout and cancellation for build jobs triggered by iCloud sync events to avoid the concurrent sync bug referenced in sync/set.js.
• Start writing tests for icloud client
• Start writing tests for macserver
• Instead of our bespoke header modifiedTime we should use 'Last-Modified' and note we don't need to set this for route using res.download – res.download does this automatically
• Rename /watch route to setup-complete
• Remove recursiveReaddirRoute
  • instead, to sync folder tree, whenever readddir is called, it should maintain an internal map of the last time a blog's subfolder tree was recursively checked, and call it? or maybe just call ls and brctl download on the specific folder? or is this too slow?
• Investigate the EHOSTUNREACH erors on the main server when attempting to communicate with the mac server – some firewall issue with mac hosting provider?
• Investigate: can we modify app/clients/icloud/routes/site/upload.js and app/clients/icloud/macserver/httpClient/upload.js to exchange a content hash? to prevent needless re-upload?
• Investigate setting up paralell server with 'Advanced Data Protection' enabled so users with this setting can sync to Blot as well

---

Fix bug with placeholder files when switching clients which will lead to data loss because Blot will not transfer the contents of placeholder files across

Fix bug with the paypal next bill dates are not syncing correctly for some users (e.g.slopserve)

Fix bug with overdue subscription payment form not working

• Possible bug introduced by recent stripe upgrade?
• Migrate Stripe library to v7

Fix bugs with custom domain setup

• Build in some of these features from the demo: https://customdomain.ai/
• Validate the domain is pointing to Blot before allowing resty-auto-ssl to request a cert. we seem to get into a DNS mismatch where a client thinks a domain points to Blot, and triggers a cert request through resty-auto-ssl but whatever DNS server Blot's server points to has the stale record (pointing elsewhere)
• Disable redirect to HTTPS for blogs by default (to ensure old browsers without SNI work) but ensure all prominent links to the blog on dashboard use HTTPS
• cache domain validity and display checkmark on dashboard

Fix bug with link format using filename

• follow up with yct

Fix bug with light mode colors

• tell Gary

Fix bug with menu which occur when multiple pages are added at once Menu needs concurrency or atomic operations (so we can build posts in parallel)

• move menu into its own model (it's possible to add thousands of entries to it...)

Fix bug with many twitter embeds

• and tell Nash

Fix bug with filtering paginated list of entries

• tell Roy and Josh and Logan

Fix bugs with template engine:

• Prevent double submission on template editor at insertion and front-end levels
• Make it possible to rename template directories and tell Darren and tell questioner and questioner
• Investigate bug with template views whose names include hashtags and follow up with Jay
• Fix bug with undeleteable template files prefixed with dot and tell Stuart
• Fix bug with template editor and ensure POST error messages are passed to client properly through ajax
  • to trigger, submit a >5mb file on web template editor
• Fix bug with chinese characters in template name and follow up with Alex
• Fix bug with date settings on the template editor for magazine template
• Fix date selector for template and tell Zach
• Fix bug with forking / editing new templates and tell Jarrod when local template folders better reflect template names

Fix bug with wordpress importer

• follow up with Simon

Fix bug with date metadata on Vincent's blog and follow up. I suspect it's to do with the ordering of date tokens.

Fix bug with potentially TeX-related escaping issues and tell Anthony and Roy

Fix issue with variable passed to dateFormat function and tell Leo

Fix bug with handling of queries and tell this person.

Fix bug with tags that contain slashes:

• tell Jack
• tell Dave

Fix bug with search for 'Raycast' and tell Anders

Fix bug and tell Vanessa which injects <p> tags

Fix bug with blockquotes and tell Kerim. Multiple ">" symbols on the blank paragraphs between other quoted text are not formatting properly.

Fix bug with indented code in pre tags and tell Michael

Fix bug with nested Markdown and tell Martin:

Fix bug accessibility issue with menu links reordering and tell Grace

Fix bug with accessibility issues with scrolling and code blocks and tell Eli if I make any changes

---

Record new video

• Could we use 11labs to make the video voiceover multi-lingual? Change my voice for someone else?
• Improve the script

Make it possible to create a site for free then require subscription to make it public

---

Follow-ups from drag-and-drop template upload:

• Move the folder browser (views/dashboard/folder/directory.html) onto the shared views/js/collect-dropped-files.js instead of its inline copy of the same traversal
• Surface template metadata.errors in the dashboard editor, not only on the preview subdomain, so a template with a broken view says so where it is edited
• Reconcile the two url defaults for a new view: source-code.js maps foo.html to /foo, readFromFolder and the upload route map it to /foo.html

New templates:

• Release Fede's theme as profile template
• Transfer theme for John

Switch templates from entries.html to index.html with explicit url: ['/', '/page/:page'].

• fix docs on routing order

Improve markdown:

• Turn on 'preserve linebreaks by default' for markdown
• Extend markdown parser to support content blocks syntax
• Extend markdown parser to add a special class to a list of only images for targeting into an image grid or gallery?
  • - ![Image](_image.jpg)... -> <ul class="image-list">... could then be used in CSS?

Improve webloc bookmarks:

• If the bookmark file points to a google doc, can we support google docs even from Dropbox? if the doc is public?

Improve images:

• Use avif/webp in responsive tags with jpg fallback for images in posts
  • use webp for image post conversion format app/build/converters/img/index.js
  • use for image caching module
  • add finer-control options for image caching / quality compression
    • Fetch dimensions of images even if optimize and cache is disable, so that image zoom can work. Tell Joshua
  • use for thumbnails
  • use for responsive images in posts / generated from image files
  • Tell Audun
  • Tell Robin
  • Implement for thumbnails too
• Add support for large image tiling and zooming with photoswipe or add a plugin using OpenSeadragon to add Deep Zoom capabilities for huge megapixel images so you could publish them directly to Blot.
• Extract multiple thumbnails per blog post (as many as there are images) for cooler image grid possibilities. Add a property to entries which lets you list all the images in a post so Jeancan add them to his sitemap
• Image cacher should work for webp, tiff, etc. as well as JPG

Improvements to docs and documentation:

• Add way in to the updates page on the index page (i.e. surface recent tasks done and to do)
• Add way in to the questions page on the index page (i.e. surface recent questions)
• Rewrite code which generates a fake 'macOS' folder to work cross platform:
  • if the user is browsing from Windows create a windows folder
  • if the user is browsing from linux create a linux folder
  • Windows video + windows finder.css
• Personalize the documentation for logged in customers
• Improve tools section of docs
  • Update obsidian tools pages (improve)
  • Improve tools/bear
    • Reach out to Bear to let them know about textbundle support
  • Improve tools/ulysses
    • Reach out to Ulysses to let them know about textbundle support
  • Create tools pages for AI assistants (Claude Code, Cursor, Codex) explaining how to use them to create templates

Improvements to template editor:

• Do preview subdomains respond to etags?
• Make preview hot reload more efficient by only reloading when the tab is visible
  • the injected script in app/blog/render/middleware.js and app/blog/views/template-error.html does new EventSource('/__blot/preview/reload').onmessage = function() { window.location.reload(); } on every message, including background tabs / hidden template-editor iframes. the expensive part is the full render that location.reload() triggers, not the SSE connection (keep that open; 10s heartbeats are cheap)
  • follow the visibility pattern in app/views/js/sync_status.js: if document.visibilityState === "hidden", set a pending flag; on visibilitychange to visible, reload once
  • debounce? no long debounce. the server already publishes once per completed sync that had changes (app/sync/index.js after the cacheID bump), so events are not per-keystroke. coalesce instead: one pending reload is enough if several events arrive while hidden or while a reload is already in flight. a short (100–300ms) debounce would only matter for back-to-back completed syncs while the tab is visible
  • etags? not nicely, not yet. location.reload() may send If-None-Match / If-Modified-Since and openresty already forwards those (proxy/config/reverse-proxy-preview.conf). render middleware sets Last-Modified from cacheID and Express default weak etags come from the body, but freshness is checked in res.send() after the expensive render, so a 304 still costs a full render. skipping the reload when hidden is the real win. a cheap cacheID-based etag checked before render would actually leverage them — related to the etag question above
• Add commit mono font to font picker
• File upload feature (_url) needs to generate image dimensions too.. maybe it needs MORE power
• Enforce size limits on setting tempalte views
• Allow text entry fields in template settings from the dashboard for template locals, and tell Omar (Build markdown description feature into template editor. Support default site description)
• Introduce collapsible sections or a separately scrollable div on the template dashboard settings, and tell Omar
• Build image upload feature into template editor. Support default banner images and image avatars, with default images set, maybe some kind of global library feature across templates...
• Build favicon generator into template editor. Make it distinct from avatar. Generate proper favicons from uploaded avatars (to prevent streched favicons when non-square avatars are uploaded)
• I wonder if we can Can we avoid storing font styles in package.json, just the ID? then inject these at render time?

Improvements to templates:

• Improve hypertext template to support listing pages as well as posts in the sidebar
• Add media-query to prevent typeset hanging punctuation on small screens (it looks weird) and tell Shibel
• Improve sidebar blog template navigation on small screens

Improve importer:

• Add way to restore a blog from export
• Update question on what to do to import posts from wordpress or squarespace: instead of contact me, reference the importer.
• Can we offload an import into a resource constrained child process?
• Can we queue up imports and process them one at a time?
• Add interface to move import into folder directly
• Add scheduled imports on regular basis into folder directly
• Add nice import step under 'Services' and as part of sign up
• Improve Ghost importer and tell Vin
• Improve Are.na importer so it handles descriptions + also other file types (e.g. PDF, text) and tell Marco
• Add micro.blog import to dashboard and tell [Gabriel](https://mail.google.com/mail/u/0/#inbox/FMfcgzGwJRvfgrRhXwqDNDlxbpXmRlbd]
• Create importer script for John's posts from flickr
• Create importer script for tumblr text files and tell Sawyer
• Create importer script for Weebly and tell David
• Map excerpt metadata to summary for Jekyll converter importer
• Look into how Hexo has written imports for blogger, joomla, rss, wordpress.
• Add an importer for Twitter and sell Sawyer

Let customers duplicate an existing site as a starting point

Improve exporter:

• Make it possible to configure the export file format (e.g. instead of markdown, word documents) on the importer
• Add export button to overdue subscription page

Blogs have huge numbers of archived templates – modify the tempalte archiver to only 'create' templates for sites using them, otherwise disappear them. work out how to clean up some of the many archived templates added to sites... (remove ones with zero changes?)

Fix script to replace cssURL to handle blogs with invalid domains (it's breaking the migration)

Replace existing cssURL and scriptURL and css_url and script_url properties in templates with new cdn helper

• do this by editing the templates
• once confirmed all templates are fixed, remove the code and documentation for these variables

Replace existing asset helper properties in templates with new cdn helper

• Check it's only used for CSS/JS...

---

Implement TOC Plugin

• Create a plugin-based TOC system that works for all file types. The plugin will generate TOC HTML from headings and return it via the result object mechanism from task 1.
• Tell Omar

Investigate replybox for comments

Add checkbox to toggle newsletter subscription to account page

Make it possible to connect two colors in the template editor, e.g. navigation background color -> same as background color

Add lang selector (map english -> en) to template editor

Create a new template local '{{{system-css}}}' which merges {{{appCSS}}}, {{{font.styles}}}, {{{title_font.styles}}}

Secure deploy ssh on GitHub

Add GitHub deploy button for macserver

Add GitHub deploy button for openresty

---

Add support for 2FA (Two factor authentication) then tell Luke, Max, Chaz, Prathamesh and Tom.

Add support for SSO then tell Prathamesh

Improve the camera make and model names extracted by exiftool using a third-party library, e.g. SONY ILCE-7M3 -> Sony α7 III

• Tell Chuck

Compute 'words' text property for each entry (unique words, sorted possibly?) to improve the speed of search queries

Identify areas of the codebase which need resource limits to be more secure, e.g. max file size for the image converter, max file size for the docx converter. look across the codebase and create a series of implementation tasks for the most pressing

Add photo/pictures/pics/images grid to tagged page of blog template and photos subfolder to default folder.

Add plugin to generate anchor links next to headings automatically and tell Peter

Refactor front end domain page to have radio button to seelct between custom domain and blot subdomain Inbox

• don't delete custom domain, though (maybe add property redirectDomain ?)

Think about how to improve the efficieny of the app/views/js/sync_status.js response from the server when loading the new contents of .live-updates. perhaps we could intercept send requests and use cheerio to filter out everything except the live update div? or some other method? and once we do that we could then leverage etags for improved performance too

Reach out to include Blot on textbundle website

Look into figma make

Can we use lightningcss to replace the existing css minifier for performance?

Transformer should short-circuit network lookups to {{config.host}} or {{cdn}} urls and pull the files directly

• and have limits on resource size and time

Add support for Apple Pay and Google Pay by upgrading Stripe API version we're using

Improve performance rendering sitemaps by using sitemap indexes, and sharding the sitemap.

• requires adding pagination for listing pages
• requires using the url / param feature, e.g. [sitemap.xml, sitemap.posts.:page.xml, sitemap.pages.:page.xml]

Speed up 404s

• Analyze req trace
• Remove case insensitive lookup and add path resolution to plugins

Improvements to template engine:

• Speed up global template build by not even invoke the setView function if there are no changes... (this avoids running updateCDNManifest when SITE template has no changes)
• Implement limits for allentries, sitemap, etc.
• Maybe we mix cacheID into query string for {{#cdn}} on preview subdomains with caching headers for improved template editor performance without expensive hash computation?
• Performance improvements
  • Limit number of posts in rss feed to 15 (modify recent entries?)
  • Add pagination to /tagged
    • should resolve issue with https://eli.li/tagged/og
• Improve various performance issues with archives page and sitemap on large sites - maybe allEntries only retrieves some properties and paginates through rather than a single request?
  • highlighted by https://jimhardy-datascience.com/search?q=2026 and https://www.nitinpai.in/archives
  • tried storing entries as a Redis hash + narrowing reads to just the fields a template references (PR #1831 unwinds it) - regressed prod performance and did not help either site. Root cause on these two sites was never diagnosed; don't retry the same approach without figuring that out first.
• why is the feed so huge? are we compressing feeds? so slow for nginx?
• finish function to render files in the user's folder dynamically and follow up with questioner
• maybe write tool to check before/after for lots of test sites? real life sites on blot? error if diff, or offer chance to test?
• Add a related entry list and tell David and Thomas and questioner and questioner
• Consider adding support for nested page structure - generate nested menu items from nested folders in /pages
• Switch from mustache to handlebars
  • Precompile using template content hash
  • Add strict verification for output differences
  • Can we generally do a ‘static’ build out directly into a cache (e.g. openresty)?
• Make it possible to query recent entries by tag and tell Peter
• Assign guid template IDs
• Assign unique path to template
• Assign template withs with full filename
• Make writeToFolder more efficient by not removing the entire folder each time (only remove orphaned files) and by no rewriting identical files
• Add search bar to template editor
• Make syntax highlight dark mode light mode compatible somehow?
• Work out why rebuilding template from folder clears cache even if the template is not changed
  • stop this
• Make it possible to source a template from a github repo (live) and tell Nash
• Add new template lambdas
  • Whitespace trimming and tell Chad
  • Number formatting (e.g. add commas to int.) and tell Chad
• Make template slug seperate from the template ID, simply for human readability
• Investigate performance of rich text editor in Safari on template editor and follow up with Rachel
• Add blog.statistics and entry.statistics (total words, etc.) and tell Pratik
• Work out how to make live but saveable modifications for preview purposes on the template editor. I have already established a messaging channel between preview HTML pages and the template editor with the script injected in render/middleware.
• Add background patterns to template editor tools, tell Ish
• Add 'group by' function which would allow you to group posts by month/ year in the same way that {{#archives}} lets you and tell Jay
• Respond to Michael's thoughtful emails about the template system
• Add support for chronological template ordering and follow up with questions
• Add way to iterate over public and private pages to allow sitemap to work properly
  • tell Rodrigo
  • questioner
  • Tell Steve
  • Jack
• Allow the name/slug of a template to change and tell Marcus and Pratik and Brad when the name of a locally-edited template in your blog's folder updates
• Expose request query to views and tell Abhishek
• Add way to query list of posts using SQL?
  • Add support for entry lists of variable length and tell Logan
• Add support for the setting of custom headers (e.g. to enable CORS) and tell Roy
• Add support for listing files in arbritary folders (e.g. use a custom folder as a 'public' folder) and tell Nash and Marco
• Add way to list directory contents and tell Diego
• Move to data directory for templates
  • Serve the template directory as a static folder, allowing things like favicon.png and humans.txt to not become blog posts. Follow up with Jamie
• Re-do url redirector to use route-parser and properly support regular expressions
• Add support for paginated list of entries on other pages and Dan tell Kevin and Eli and Sven and Nikhil and Adam and Madino and Ryan
• Add dynamic querying of entry list and tell Naoki
• Add a popular entry list
• Add way to auto-refresh page during template development on preview subdomains. Tell Ryan
• Add way to list files at a path and tell Matt and Daisy
• General:
  • Add paginateable entries list
  • Add downloadable sites
  • Add static file uploader
  • Make templates folder based?
    • data/templates/blog_id/template_name
  • Add webfont uploader
  • Generate favicons of different sizes from uploaded image
• Re-enable partial templates in posts/pages
  • Tell Matt
• Add pagination to archives and tagged pages on default templates using a new tag
• Inject title metadata into h1 tag for all the text-based default templates and tell questioners
• Add 'download as zip' option to template editor
• Create starter kits for each template
  • e.g. a starter kit on the blog template for a writer, or a photographer, or a journalist, or an academic.
  • these could be downloaded as folders on the template page?
  • Get David's feedback when one exists for writers
• Create an index of customer-created templates and add Matt's theme
• Add 'Hidden: yes' metadata to replace 'Menu: no' and tell Sebastien
• Add method to merge updates into custom templates and tell Dave
• Redirect URLs with trailing slash when possible (i.e. not directory browsing) to fix possible duplicate content penalties and tell Marius
  • Add rel=canonical to all the default templates, or resolve the problem that Blot serves the same thing to URLs with a trailing slash, and without.
  • If we do add rel=canonical, make sure to consider the situation where someone mounts a page to the index of their site, e.g. /page/1 and / will now be different.
• add support for dynamic partials to make collections possible, e.g. {{> /:tag.html}} ?
• Add neat interface to customize blog level variables?
• How do we transition from a global date display format to one that templates have control over?
• Consider making file-based template directory a git repo and using this to implement basic undo-redo functionality.
• Disable re-rendering for locals
• Add better customization options
• Local template editing does not clear cache
• Use Hogan for precompiling and rendering templates for performance?
• Parse tokens and params from routes and tell Jason
• Add way to allow all entry lists to access {{#months}} sorting options just like the archives page.
• Tell Marco when arbirary list sorting is ready
• Add support for fetching only some properties of an entry (ideally only those that are used). prevent some properties from being fetched in some lists (html, for example in all_entries). Would be a good candidate for hgetall or hget?

Launch new server and test that the new scripts to mount the instance store and data ebs-disk work

Automate the detection and removal of long-overdue sites (> 1 year)

• Prevent long-overdue customers from accessing the service, then tell Matvey

Implement toString for all existing template lambda, see retrieve/updated.js for an example

• Map {{date}} => {{#date}}...{{/date}} and {{#updated}} and {{#created}} etc.
• Update developer docs
• Can we use this technique elsewhere? Simple string vs. function...

Improve folder to CDN pipeline

• Could we respond with redirects to Blot's CDN?
• Why isn't the WAV file in this footnote being properly replaced with a CDN URL?
  • it's because it contains URL-encoded chars – replace
  • https://diwank.space/field-notes-from-shipping-real-code-with-claude
• Why isn't this file?
  • https://yangrobotics.com/posts/peng-2.md
  • possibly because it's a fully-qualified URL? we need to handle these I think
• Maybe all 'asset requests' should return 302? even 301 if the url is versioned?

Move image zoom plugin to templates

• Fix bug with image zoomer on mobile devices and tell Alessandro
• Fix bug with images zooming inside footnotes and tell Nathan
• Document image zoom disable feature ?zoom=false and tell questioner For each blog: if image zoom plugin enabled: for each template: inject required CSS just above {{{appCSS}}} and save inject required JS just above {{{appJS}}} and save disable image zoom plugin Remove image zoom plugin

Reduce service provider risk:

• move domain registrar from AWS to Epik?
• Remove dependence on AWS
  • Adds offsite (i.e. non AWS backups) backups
  • Write out plan to move off AWS
    • DNS
    • EC2
    • S3 for backups
    • Cloudfront for CDN
    • backup blogs folder + db somewhere offsite at least once per week
• We should have a plan to switch CDN from Bunny to Cloudfront in an emergency
• Should be a simple DNS change
• write plan to transition off Bunny if needed
  • ideally all this should involved is changing some DNS records
• test plan to transition off Bunny
• write plan to transition off AWS if needed
  • ideally all this should involved is changing some DNS records
• test plan to transition off AWS

Update redis client for improved performance

• perhaps to ioredis?

Move folder lock location to instance ssd for improved reliability and performance (non-networked drive...) – could be inside the cache

Extend ignore files feature (currently files starting with _ or folders starting with _)

• tell Troy
• Follow up with Shibel about how to ignore - make it possible to achieve through a global 'ignore' setting on services page of dashboard?
• could ignore all '.jpg' say, or all 'index.html' files, etc.
• Can we add a 'folder rules' section to create ignore globs, multi post globs, etc. arbritrary paths and keep clean URLS (no underscores). Sitewide .blotignore file?
• handle new pathNormalizer behaviour
• Add Ignore: yes metadata to trigger the ignored file behaviour

Think about how to purge the old versions of files cached by bunny's permacache

Reduce size of css file by only shipping katex css once the site first publishes a post with katex

Testing:

• Write unit tests for database models which have no tests subdirectory
  • e.g. models/404, models/entries, models/redirects
• Add tests for webp thumbnails
• Add integration tests for cdn paths with weird view names, etc.
• Add integration tests for preview subdomain logic (skip for my- not for site)
• Add end-to-end tests for brochure site to test sign up flow (use Playwright? Phantom? Chrome headless)
• Add spell checking test
• Write browser based integration tests for core functionality
• how can we test the 'reload-server' script?
• how can we test the 'scheduler' functions? we ran into error where it was running twice

Resolve bad links (e.g. incorrect case) in files at build-time rather than run-time so we can avoid the expensive disk lookups in blog/assets.js

Add a highlevel function which clients can expose to open a file (e.g. link to dropbox.com or google drive)?

Run image analysis on all images in post

• extract colors for searching
• use something to identify related images

Improvements to questions section:

• Add 'blue dot' notifications icon to updates and questions based on last post
• Add colors to tags on questions section:
  • https://gauthierroussilhe.com/articles
  • https://designportland.org/
• Add a filter for topic index to show questions by: 'Newest', 'Active', 'Unanswered', 'Votes'

Add auto template translator

Add global site search of:

• files on site
• posts on site
• documentation
• questions
• templates

Improvements to local folder experience

• Interpolate fonts based on an ID (a la web template editor)
• Interpolate syntax highlighting based on an ID

Surface recently edited posts on the sites tab of the dashboard

Can we support white-label cdn subdomain so there are no Blot CDN links?

Write tests for animated gif thumbnail

Surface Blot client sync errors on dashboard

• add email notifications for major sync errors (i.e. dropbox token revoked)
• surface dropbox out of space error
• Expose markdown build errors on the dashboard to prevent weird non-update situations where it looks like the sync is stuck

---

Preserve single line breaks in Word Docs for poetry and tell Isabel and tell Paul

Purge thumbnails and cached images which are no longer in use

For broken embedded images, attempt to resolve based on a shortest path approach, as in, traverse the folder and attempt to find an image with a matching name

• Tell Ellane

---

Embed feature

• Add support for native PDF embeds in posts, just put a link to a PDF or a path, e.g. /_assets/file.pdf --> would become a nice embed.
• Make it possible to embed PDF like so ![Video](...)
• Make it possible to embed video like so ![Video](...)
• Add support for native MP3 embeds in posts, just put a link to an MP3 or a path, e.g. /_assets/file.mp3 --> would become a nice embed then tell Luke and Nash and Andy and Charlie and Konstantin
• Add soundcloud embeds then tell Jonas, Nashp, and Anders and Richard
• Add are.na embeds and tell Marco

Improvements to search engine:

• Make search use 'AND' instead of 'OR' for multiple terms
• Remove drafts and tell Paul
• Make search: true work and tell Paul

Rewrite broken link tests

Add pause subscription for Stripe and PayPal

• Enable hibernation of blogs

Add disclaimer to pricing page for old subscribers

Add Generative tree for account page

Add {{#posts-without-:tag}} and tell steve

---

Ensure that overdue PayPal subscription webhooks map to notification emails

Clean up CDN permanent storage:

• Purge old docs CSS and JS from CDN storage cache

Make sure main server can hard restart without issue

Data use minimization:

• Write automated script to archive overdue blogs from disk and db
• Write script to remove unused and unaccessed static files from within blog folders
  • we'll also need to purge the cache of the transformer module for the file

When a post is deleted, we should remove everything except the created time?

Create a queue for purging cache, group requests together to prevent stampede

Prevent overzealous purging of cache for syncs in which the folder does not change

Improve paypal integration:

• Get create new blog working
  • Add way to modify subscription quantity for paypal users
• Create new router for paypal subscription settings
• Test subscription with real paypal account
  • Test new blog creation
  • Test new blog deletion
  • Test switch to annual billing
  • Test account deletion
• Make it possible to add paypal as a payment method for existing stripe users
• Verify signature integrity of paypal webhooks
• Use PayPal buttons api to modify subscription properties
• Get account page working
  • Add way to cancel subscription from dashboard
  • Add way to delete account from dashboard
• Update scheduleEmailNotification for paypal customers

Make {{active}} strip the domain from the link before checking? Or automatically strip the domain from internal links on Blot's dashboard?

Store path -> md5 hash on sync

• Use to auto version links to files in your folder so they can be cached

---

Adjust auto titles of 'screenshot at 12:22pm' and make sure the date parser can handle them

Adjust title generator so h2 titles within the file dont become the title of the post – this is helpful for obsidian

---

Server improvements:

• Copy the cron tab into the new server setup script
• Add more stringent checks to scripts which mount instance storage and external volumes
• Test reboot for all servers

Add support for proxying particular analytics files and tell questioner

One bad latex equation should not tank the build process for the whole post

Add support for activitypub and tell

• Tim
• Questioner
• Rhys

Methods to improve server performance

• consider expanding the replication of the perma_cache across more regions
• consider switching to more expensive CDN type (i.e. not volume)

---

Add support for paying with crypto, BTC ETH

• Consider https://github.com/btcpayserver https://github.com/btcpayserver/node-btcpay
• Tell Chaz
• Tell Anita
• Tell Arc
• Tell Dmitriy

Improvements to server

• test maintenance and redirector on dashboard
• restrict post to /dashboard route so we can tune nginx logic?
• tune performance of express-mustache in production and development
• add optional HTML/JS/CSS minification to express-disk-cache
• Test overdue subscription re-start
• Prevent GET request to log in page from setting connect.sid cookie
  • or otherwise work out how to try and cache everything at nginx
  • log in route should clear 'logged_in_to_blot' cookie if not logged in
• Use Lory to add a slider to the homepage

Improvements to sync/reset scripts:

• reset should produce a report which I get email

---

Add way to escape commas in tags and tell questioner

---

Add simple form endpoint, could be used for contact forms

Add theme with photoswipe and use sharp's tile generation feature along with

Add way to set the slug-part of a URL for a post and tell George and follow up for post in Questions and tell Jarrod

---

Investigate migrating from Redis to SQLite per-blog database

• Evaluate feasibility of storing one SQLite database per blog in data/blogs/{blogID}/blog.db
• Assess performance implications (memory usage, query speed, connection management)

Can Blot blogs be federated? Respond to Nigel once I do some research on this

---

Add tag to iterate over all images in a folder and tell Laurel

---

Make it possible to link between org mode files with: [[file:another-test.org][another test]] and tell questioners

Improve backlink's tolerance

• Add support for full qualified URLs in the backlinks list for each entry. For example, if your blog is example.com, the following link will not appear as an internal link: Foo because build/prepare/internalLinks is not aware of the blog domain/handle
  • Tell Eli
• Add support for internal links without a leading slash
  • Tell Laurel

---

Move citations out of {{{body}}} and into a new tag {{{citations}}} and tell questioner and Mikka? maybe more structured, so you could also do: {{#citations}}...{{/citations}}?

---

Improvements to teaser generator

• Improve estimate of teaser length
• Remove trailing non-text elements (like

  ---

  )
• Add {{small}} and {{long}} versions of teaser?
• Internal links (e.g. #section) in {{{teaser}}} should be resolved against the entry's permalink.
• Add hasBreakpoint property to entry to allow greater control over display of {{{teaser}}} and tell Pratik
• Make it possible for users to select length of teasers and Leo

Make it possible to disable draft preview files

Performance improvements

• Faster server, with SSL termination across CDN
• Use the --trace-sync-io command-line flag to determine if Blot is using any sync methods inappropriately
• Can we keep the pandoc child_process running and pipe stuff in and out for speed? Measure before after to ensure this helps
• Add request level timeouts to prevent server swamping
  • ensure to remove the timeout from the SSE streaming routes (drafts, sync status on dashboard)
  • read this guide: http://expressjs.com/en/resources/middleware/timeout.html
  • it seems we also need to cancel the resources involved in responding to the request (i.e. disk io, db calls) for this to be really useful
  • Add a timeout of 15 seconds for building posts.

Add line numbers to highlight.js hljs code tags

Convert Vitepress to Blot and tell Nicolas

Expose log files on blog's dashboard and tell Roman, Sönke and questioner and John

• Wikilinks should have a class property of 'wikilink' rather than a title attribute and tell questioner
• Improve wikilinks lookup algorithm:
  • Add support for 'shortest path where possible' on Obsidian
    • search current directory for filename
    • search subdirectories for filename
    • search parent directories for filename
  • Add support for 'absolute path' on Obsidian when the vault is a sub directory of your blog folder
    • How best to accomplish this?
  • Add support for deleted files (e.g. [[Target]] -> target.txt still works if target.txt is renamed to foo.txt)

---

Improvements to metadata extractor:

• Adjust summary generator such that a subtitle is used for summary text (i.e. an h2 directly following an h1) We still want to prevent the scenario in which a later h2 becomes the summary, though. Notify Jessica and rebuild Jessica's site
• Add support for entry property with escaped double quotes, tell Jay

Improve questions section of site, in approximate order of importance:

• Add way to mass-rename tags
• Add display aliases for tags e.g. how -> How to use Blot, etc.
• Store version history of the question and answer edits
• give the 'ask' and 'answer' textarea inputs the full functionality of the GitHub issue/PR comment inputs (i.e. code snippets / preview / copy/paste screenshots from clipboard)
• email notifications (There's an existing system in place for sending emails in app/helper/email)
• upvoting for topics and upvote-based ranking on the topic index
• show vote count and answer count next to each question on index
• 'run code snippet' feature / similar to trinkets.io

Rewrite transformer (in app/helper/transformer)

• Speak HTTP properly, respect backoffs, use/send eTags
• Share information across blogs?
• Improve transformer so it checks a file's MTIME against a stored value before rehashing the file?
• Adjust interface to accept a number of directories. Make it 'unaware' of blog IDs?
• Add method which accepts a path or URL and flushes what is stored for it, update the rebuild entry script
• Cache errors. When a post contains multiple identical links to images that timeout/404, the thumbnail generator should only make 1 attempt to fetch them. Right now we have a post that is filled with identical broken image links (in this case to a spacer.gif) can suck up the build process for hours.

Add a 'page link format' option to the dashboard – the link format now only applies to entries. Tell Mike

Improvements to server infrastructure

• Work out way to recover from expired cert in the cache. To reproduce, consider a member who signed up, connected example.com, then moved the domain to another service. The attempt to renew the cert for example.com will fail. When the customer re-registers with Blot, for some reason open-resty-auto-ssl will serve an old certificate. I followed these steps to reset but we should automate this
• Create DMARC record for Blot https://en.wikipedia.org/wiki/DomainKeys_Identified_Mail https://docs.aws.amazon.com/ses/latest/DeveloperGuide/dmarc.html

Improvements to entry builder

• Improve the automatic title generated for screenshots with default MacOs filename from 'at 14:40pm'
• Generate a list of titles which you could use to generate a table of contents and tell Zach
• Use time draft was moved into blog folder for entry created/publish date, rather than the time the draft file itself was created. Tell Kerim and Shibel
• Parse Word Document subtitles as h2 and tell Matt
• Add way to filter entries by post type (e.g. photo) and tell Justin.
• If entries share publish date, sort them by blot creation date

Improvements to dashboard

• Add error messages on overdue subscription payment form
• Use <select> optgroup on timezone select dropdown to subdivide zones by current time, and steal the presentation format used by basecamp.
• Add a broken link checker
• Indicate that some posts' permalink will not change (if metadata) on the dashboard when setting the blog's permalink format. Tell Chet.
• Dumb quotes are still smartified, even without typeset enabled, possibly by pandoc?
• Undo option for form changes
• Add transactional emails notifying customer of changes to their email or password?
• When a customer pastes the whole tracking script into the preferences page on the dashboard, strip out everything except the ID which we need... perhaps create a page explaining what is happening before doing this.
• Handle noscript on dashboard / signup gracefully (remove all dashboard js?)
• Allow customers who are restarting an account with multiple blogs to disable some before restarting.

---

Add support for paths to static files with a trailing slash then tell Sean

Finish template translation offers

• Anders Norén
• Radek Kozieł
• Port Jane hugo theme to Blot and tell Jack and Josh on micro.blog
• Ghostwriter and tell Simon
• Work on Tim's template
• Translate this theme to Blot and tell Naveen
• Translate Ana's designs to Blot
• Translate 2019 to Blot and tell Gabriel
• Translate Paul's template into Blot
• Add a bootswatch template and tell Michael
• Port the jekyll template Neocactus to Blot and document the process then tell Robert

Add way to filter archives page by year or month and tell Chad

Adjust length of slug to use as few words as possible until collision and tell Chet

---

Improve 404 log page on Dashboard

• Make the 404 page more useful (see Netlify site)
• Add button to clear 404 log and tell Jamie
• Add frequency indicator for each route 404 log and tell Jamie
• Add filtering by date to 404 log and tell this person

---

Write new app/build/converters to turn new file formats into posts

• Add support for posts generated from Excel files, would generate a table. Use this library?
• Add AVIF image post support and tell Simon
• Add support for Ulysses .ulz file format and tell Nash and add support for .ulyz file created by Ulysses Adapt this script
• Add support for Jupyter Notebook .ipynb using Pandoc and tell Arnab and notebooks (via pandoc) and tell questioner
• Add support for .doc using libreoffice and tell Kyle
• Video posts (.mp4) - add way to opt-in to these
• Add support for .tiff images
• OPML files and tell John
• Audio posts (.mp3 for now?) and tell Kevin - add way to opt-in
• Add support for INI files and tell Frank

Write new clients to sync blog folders

• Add a generic 'use your ai agent' client which lets you copy paste a prompt that configures a site with the git client behind the scenes
• Add OneDrive client
  • Tell Nenad
  • Tell John
  • Tell Shantesh
  • Tell Nathan
  • Tell ekskog
  • Tell John
  • Tell Simon
  • Tell Andrew
  • Tell Naveen
  • Tell Emad
  • Tell John
  • If OneDrive for Business is supported tell Josh
• Add Apple Notes client
  • Tell Håvard
• rsync
  • Tell Harry
• SFTP
  • Tell Coraline
  • Tell Utkorg
  • Tell Calvin
  • Tell Fabrice
• NextCloud
  • Tell Vincent
  • Tell Coraline
  • Tell Andrew
  • Tell Mike
• Box
  • Tell Eric
• Syncthing
  • Tell Jason
• Github
  • Tell John
  • Make it possible to connect github repo via their API and tell reg
• Jumpshare
  • Tell Ömer
• pCloud
  • Tell Omaid
  • Tell Anders
  • Tell Tim
  • Tell JMN
  • Tell Rodney
• Dropmark
  • Tell Rado
• Proton Drive
  • Tell Ted
  • Tell Francois
• WebDav

---

Add password protection for blogs

• tell Ed, this person, this person, and Will, and Modomiro and Julian and Raph and Jeremy and Tara and Phil and Florian and Roy's friend Tim and Kaleb and Lotus and Khaled and Noel and Franz and Justin and Kyle and Devan and John and Anita and Emerald and Ben and Giuseppe and Annie and Jan
• think about how to add simple password (just a string) protection to sites.
• we need to consider both the node.js app (app/index) and the openresty reverse proxy (proxy/), especially the proxy caching behaviour for sites.
• perhaps we'll need to handle sessions for sites with password protection in some way at the open resty level? or just use some response header from upstream to always bypass the proxy cache? or some cookie? we don't want to add any extra work in openresty for sites without passwords. perhaps sites with passwords return responses with a header that prevents any proxy caching from taking place?
• anyway, it would involve:
• updating the blog model (password: string).
• adding a simple on/off input button to the dashboard, possibly under a new section.
• creating a password form view common to all sites, or able to be overwritten (password.html in a template perhaps).

Add server-side analytics:

• Tell Tom
• Tell John
• Tell Romain

Add commenting system

• Tell Precious
• Tell Hochan who is willing to beta test

Add email newsletter system

• Tell David
• Tell Nash
• Tell Kyle

Add paid subscription system

Create an API for Blot

• Add webhooks and tell Aurynn and Shawn
• Use it to build a minimal blot cli sync client run from the command line (could background itself?!)

---

Self-hosting:

• Write self-hosting documentation
  • Tell Jake
  • Tell Berndl
  • Tell Joel
  • Tell Mohammed
  • Tell BP
  • Tell Rado
  • Tell Nuno
  • Tell Omaid
  • Tell Brian
  • Tell Jarod
  • Tell Francesco
  • Tell Pranay
  • Tell Aryan
  • Tell Todd
  • Tell Anita
  • Tell Kaushik
  • Tell Michael
  • Tell Frank
  • Tell Zip
• Tell Nick when it's possible to deploy a self-hosted Blot on Vercel and Zia

---

Flaky specs (newest first; see "Flaky specs" in AGENTS.md). Remove an entry when its fix merges.

• 12 Nov 2026, app/helper/lock/tests/lock.js "helper/lock lock treats a release retried after the key expired naturally as released": Error: Lock acquire reply too slow: test:helper-lock:lock:... at helper/lock/index.js:114 (spec line 375). Branch claude/keynote-template-audit-26b7aa, 69b18f6d9 (#2680), which only touches the Keynote template. Job: https://github.com/blotcms/blot/actions/runs/43237684899/job/131249356200. Investigate: the acquire-reply deadline is wall-clock and a busy CI lane can exceed it; give the spec a longer reply timeout, or make the deadline injectable for tests.
• 7 Nov 2026, CI test setup (no spec): node.yml test 3's lanes app/clients/git 1/2 and app/helper 1/2 died before running a spec, right after "Setting up CDN on cdn.localhost", with TimeoutError: @smithy/node-http-handler - the request socket did not establish a connection ... within 5000 ms. Branch claude/flaky-welcome-email-claim, 4f9c3a361 (#2664); passed on re-run. Job: https://github.com/blotcms/blot/actions/runs/42399824750/job/128484988122. Investigate: the local S3 endpoint the CDN setup calls may not be ready when a lane starts; wait for it, or retry the setup call.