Merge pull request 'Add strfry policies for the public submission relay and approved sync' (#267) from feature/submission_relay into master
CI / Test (push) Successful in 2m8s
CI / Test (push) Successful in 2m8s
Reviewed-on: #267
This commit was merged in pull request #267.
This commit is contained in:
7 files changed
+929
No files matched your search
@@ -7,6 +7,12 @@ akkounts, so that notes published to the relay have to be authored by (or in
|
||||
some cases just related to) local users who have verified their Nostr public
|
||||
key.
|
||||
|
||||
It holds both Deno policies (the members relay's write policy, the member
|
||||
bridge, and the manual sync tool — described below) and plain Node ESM policies
|
||||
for the public submission relay and the approved-content sync. See
|
||||
`extras/strfry/README.md` for details on the policies and their environment
|
||||
variables.
|
||||
|
||||
### Requirements
|
||||
|
||||
[Deno](https://deno.com/) needs to be installed on the machine that you run
|
||||
|
||||
@@ -0,0 +1,198 @@
|
||||
# strfry policies
|
||||
|
||||
Policy plugins for the Kosmos [strfry][strfry] relays. There are two runtimes in
|
||||
this directory:
|
||||
|
||||
- **Deno** — the members relay's write policy and member bridge, plus the manual
|
||||
member-note sync. Dependencies are pinned in `deno.json`/`deno.lock`. See
|
||||
`docs/dev/nostr.md` for the Deno setup.
|
||||
- `strfry-policy.ts` — the relay's own write policy.
|
||||
- `ldap-only-policy.ts` — the member bridge's `pluginDown`.
|
||||
- `ldap-policy.ts` — shared LDAP membership module used by both of the above.
|
||||
- `strfry-sync.ts` — manual member-note sync tool.
|
||||
- **Node** (plain ESM `.mjs`, Node 24 LTS, no dependencies) — the submission
|
||||
relay and the approved-content sync.
|
||||
- `submission-policy.mjs`
|
||||
- `sync-policy.mjs`
|
||||
|
||||
In development the whole directory is mounted at `/opt/strfry` (see the
|
||||
`strfry` service in `docker-compose.yml`, and `docker/strfry/strfry.conf` for
|
||||
the write-policy wiring); in production the ops tooling deploys it to the relay
|
||||
hosts.
|
||||
|
||||
## How strfry invokes a policy
|
||||
|
||||
strfry spawns the configured `relay.writePolicy.plugin` command and talks to it
|
||||
over stdin/stdout. Each line on **stdin** is a JSON object:
|
||||
|
||||
```json
|
||||
{"type":"new","event":{...},"receivedAt":1700000000,"sourceType":"IP4","sourceInfo":"203.0.113.7"}
|
||||
```
|
||||
|
||||
`sourceType` is one of `IP4`, `IP6`, `Import`, `Stream`, `Sync`, `Stored`.
|
||||
`sourceInfo` is an IP address (for `IP4`/`IP6`) or a relay URL (for
|
||||
`Sync`/`Stream`).
|
||||
|
||||
For every input line the plugin must write one line of JSON to **stdout**:
|
||||
|
||||
```json
|
||||
{"id":"<event id>","action":"accept|reject","msg":"..."}
|
||||
```
|
||||
|
||||
`msg` is only meaningful for `reject` and is returned to the client. Diagnostics
|
||||
go to **stderr** and end up in the strfry log.
|
||||
|
||||
## `submission-policy.mjs`
|
||||
|
||||
Write policy for the **public submission relay**. Open to all authors — there is
|
||||
**no LDAP/membership check**. It applies only the "usual gates":
|
||||
|
||||
1. `type` must be `new`.
|
||||
2. Event must be well-formed (`pubkey`, integer `kind`, `tags` array).
|
||||
3. `kind` must be in the allowlist (deny by default).
|
||||
4. `content` must be within the byte bound.
|
||||
5. Tag count must be within bounds.
|
||||
6. `p`-tag count must be within the hellthread bound.
|
||||
7. Per-IP token bucket (keyed on `sourceInfo`).
|
||||
8. Per-author token bucket (keyed on `event.pubkey`).
|
||||
9. Short-window content dedupe (sha256 of `content`, skipped for content shorter
|
||||
than the configured minimum so empty kind-3/kind-5 content is not falsely
|
||||
deduped).
|
||||
|
||||
All state is **in-memory** and resets whenever strfry reloads the plugin (e.g.
|
||||
on file mtime change or config change).
|
||||
|
||||
### Environment variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `SUBMISSION_KINDS` | `0,3,5,360,30360` | Comma-separated kind allowlist. Empty value denies every kind. |
|
||||
| `SUBMISSION_MAX_CONTENT_LENGTH` | `16384` | Max `content` size in UTF-8 bytes. |
|
||||
| `SUBMISSION_MAX_TAGS` | `200` | Max number of tags per event. |
|
||||
| `SUBMISSION_MAX_P_TAGS` | `10` | Max number of `p` tags (hellthread cap). |
|
||||
| `SUBMISSION_IP_BURST` | `120` | Per-IP bucket capacity (burst). |
|
||||
| `SUBMISSION_IP_REFILL_PER_SEC` | `5` | Per-IP sustained refill rate. |
|
||||
| `SUBMISSION_AUTHOR_BURST` | `10` | Per-author bucket capacity (burst). |
|
||||
| `SUBMISSION_AUTHOR_REFILL_PER_SEC` | `0.2` | Per-author sustained refill rate. |
|
||||
| `SUBMISSION_DEDUPE_WINDOW_MS` | `60000` | Dedupe window. |
|
||||
| `SUBMISSION_DEDUPE_MIN_LENGTH` | `50` | Minimum content length that is deduped. |
|
||||
|
||||
### strfry configuration
|
||||
|
||||
```ini
|
||||
relay {
|
||||
writePolicy {
|
||||
plugin = "node /opt/strfry/submission-policy.mjs"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## `sync-policy.mjs`
|
||||
|
||||
Write policy for the **approved-content sync** on the members relay. It runs
|
||||
under a separate strfry config that shares the relay's database, and it is
|
||||
**fail-closed** and does **source verification only**; all content gating
|
||||
(kinds, sizes, rates, …) is handled by the other policies. An event is accepted
|
||||
only when *all* of the following hold:
|
||||
|
||||
- `type === "new"`;
|
||||
- `sourceType === "Sync"`;
|
||||
- `sourceInfo === STAGING_RELAY_URL` (byte-exact).
|
||||
|
||||
Everything else is rejected, and the log line records the **observed vs
|
||||
expected** `sourceType` and `sourceInfo`. If `STAGING_RELAY_URL` is unset, the
|
||||
policy rejects every event.
|
||||
|
||||
### Environment variables
|
||||
|
||||
| Variable | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `STAGING_RELAY_URL` | _(unset)_ | Expected `sourceInfo` for synced events, e.g. `ws://staging-relay:7779`. Unset/empty rejects everything. Whitespace around the configured value is trimmed; `sourceInfo` is compared as-is. |
|
||||
|
||||
### strfry configuration
|
||||
|
||||
```ini
|
||||
relay {
|
||||
writePolicy {
|
||||
plugin = "node /opt/strfry/sync-policy.mjs"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The matching sync is driven by `strfry --config <sync-config> sync
|
||||
ws://<staging-relay>:7779 --dir down`, which sets `sourceType=Sync` and
|
||||
`sourceInfo` to the relay URL passed on the command line. `STAGING_RELAY_URL`
|
||||
must be that exact URL.
|
||||
|
||||
## `strfry-policy.ts`
|
||||
|
||||
Deno write policy for the members relay itself. It combines the hellthread cap,
|
||||
anti-duplication (via Deno KV with a 60s window), and the shared `LdapPolicy`
|
||||
membership check, but lets NIP-46 (kind `24133`) ephemeral events through from
|
||||
anyone. See `docs/dev/nostr.md` for the Deno requirements and configuration.
|
||||
|
||||
## `ldap-only-policy.ts`
|
||||
|
||||
Deno write policy for the members relay's **member bridge** — a `strfry router`
|
||||
stream group (`dir = "down"`) that subscribes to the local submission relay and
|
||||
writes member-authored events into the members relay's database. It is this
|
||||
stream group's `pluginDown`, and it accepts an event **only** when its author is
|
||||
an LDAP member (or a whitelisted pubkey).
|
||||
|
||||
It reuses `ldap-policy.ts` and the shared `.env` (the same read-only LDAP
|
||||
service account as `strfry-policy.ts`), but deliberately does **no**
|
||||
hellthread/anti-duplication work and opens **no** Deno KV store — so it can run
|
||||
alongside the relay's own policy process without shared state.
|
||||
|
||||
### strfry configuration
|
||||
|
||||
```ini
|
||||
streams {
|
||||
members {
|
||||
dir = "down"
|
||||
filter = { "kinds": [0, 3, 5, 360, 30360] }
|
||||
urls = [ "ws://<submission-relay>:7778" ]
|
||||
pluginDown = "/opt/strfry/ldap-only-policy.ts"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
driven by `strfry router <member-bridge-config>`. `dir = "down"` keeps it
|
||||
strictly one-way (the router never watches or forwards the local DB). The
|
||||
`filter` bounds the LDAP lookups to the submission kinds; membership is decided
|
||||
per event by the policy, so no member list is maintained.
|
||||
|
||||
## `ldap-policy.ts`
|
||||
|
||||
Shared Deno module exporting `LdapConfig` and the `LdapPolicy` class: it looks
|
||||
up the event author's `nostrKey` in LDAP (with a pubkey whitelist and special
|
||||
handling for zap receipts) and rejects non-members. Imported by both
|
||||
`strfry-policy.ts` and `ldap-only-policy.ts`.
|
||||
|
||||
## `strfry-sync.ts`
|
||||
|
||||
Deno script for one-off/manual syncs of all notes authored by local users (any
|
||||
account that has verified its Nostr pubkey with akkounts) from a remote relay via
|
||||
negentropy sync. See the "Manual tasks" section in `docs/dev/nostr.md`.
|
||||
|
||||
## Tests
|
||||
|
||||
Unit tests use the built-in `node:test` runner (no dependencies):
|
||||
|
||||
```sh
|
||||
node --test extras/strfry/test/*.test.mjs
|
||||
```
|
||||
|
||||
## Smoke test
|
||||
|
||||
```sh
|
||||
printf '%s\n' \
|
||||
'{"type":"new","sourceType":"IP4","sourceInfo":"1.2.3.4","event":{"id":"a","pubkey":"b","kind":360,"tags":[],"content":"hi"}}' \
|
||||
| ./submission-policy.mjs
|
||||
|
||||
printf '%s\n' \
|
||||
'{"type":"new","sourceType":"Sync","sourceInfo":"ws://staging-relay:7779","event":{"id":"a","pubkey":"b","kind":360,"tags":[],"content":"hi"}}' \
|
||||
| STAGING_RELAY_URL='ws://staging-relay:7779' ./sync-policy.mjs
|
||||
```
|
||||
|
||||
[strfry]: https://github.com/hoytech/strfry
|
||||
Executable
+31
@@ -0,0 +1,31 @@
|
||||
#!/bin/sh
|
||||
//bin/true; exec deno run -A "$0" "$@"
|
||||
//
|
||||
// LDAP-only strfry write policy for the production relay's member bridge.
|
||||
//
|
||||
// The member bridge is a `strfry router` stream group (dir = "down") that
|
||||
// subscribes to the local submission relay and writes member-authored events
|
||||
// into the production database. This policy is its `pluginDown`: it accepts an
|
||||
// event if and only if its author is an LDAP member (or a whitelisted pubkey),
|
||||
// and rejects everything else.
|
||||
//
|
||||
// Unlike strfry-policy.ts it deliberately does NO hellthread/anti-duplication
|
||||
// work and opens NO Deno KV store, so it can run alongside the relay's own
|
||||
// policy process without any shared state. See README.md.
|
||||
|
||||
import { strfry } from 'jsr:@nostrify/strfry';
|
||||
import { LdapConfig, LdapPolicy } from './ldap-policy.ts';
|
||||
import { load } from "https://deno.land/std@0.224.0/dotenv/mod.ts";
|
||||
|
||||
const dirname = new URL('.', import.meta.url).pathname;
|
||||
await load({ envPath: `${dirname}/.env`, export: true });
|
||||
|
||||
const ldapConfig: LdapConfig = {
|
||||
url: Deno.env.get("LDAP_URL"),
|
||||
bindDN: Deno.env.get("LDAP_BIND_DN"),
|
||||
password: Deno.env.get("LDAP_PASSWORD"),
|
||||
searchDN: Deno.env.get("LDAP_SEARCH_DN"),
|
||||
whitelistPubkeys: Deno.env.get("WHITELIST_PUBKEYS")?.split(',')
|
||||
}
|
||||
|
||||
await strfry(new LdapPolicy(ldapConfig));
|
||||
Executable
+241
@@ -0,0 +1,241 @@
|
||||
#!/usr/bin/env node
|
||||
// strfry write policy for the public Kosmos submission relay.
|
||||
//
|
||||
// This relay is open to all authors (no LDAP membership check). It applies only
|
||||
// the "usual gates": a kind allowlist, content/tag bounds, per-IP and
|
||||
// per-author token-bucket rate limits, a short-window content dedupe, and a
|
||||
// hellthread (max p-tag) cap.
|
||||
//
|
||||
// Reads strfry JSONL input messages from stdin and writes one
|
||||
// {"id","action","msg"} JSON object per line to stdout.
|
||||
//
|
||||
// All state is in-memory and resets whenever strfry reloads the plugin.
|
||||
//
|
||||
// See README.md for the env vars and how strfry invokes this.
|
||||
|
||||
import readline from 'node:readline';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
|
||||
const DEFAULTS = {
|
||||
kinds: '0,3,5,360,30360',
|
||||
maxContentLength: 16384,
|
||||
maxTags: 200,
|
||||
maxPTags: 10,
|
||||
ipBurst: 120,
|
||||
ipRefillPerSec: 5,
|
||||
authorBurst: 10,
|
||||
authorRefillPerSec: 0.2,
|
||||
dedupeWindowMs: 60000,
|
||||
dedupeMinLength: 50,
|
||||
};
|
||||
|
||||
const NAME = 'submission-policy';
|
||||
|
||||
function parseKinds(raw, fallback) {
|
||||
const value = raw === undefined || raw === null ? fallback : raw;
|
||||
const kinds = new Set();
|
||||
|
||||
for (const part of String(value).split(',')) {
|
||||
const trimmed = part.trim();
|
||||
if (trimmed === '') continue;
|
||||
|
||||
const n = Number(trimmed);
|
||||
if (Number.isInteger(n)) kinds.add(n);
|
||||
}
|
||||
|
||||
return kinds;
|
||||
}
|
||||
|
||||
function parseNumber(env, key, fallback) {
|
||||
const raw = env[key];
|
||||
if (raw === undefined || raw === null || String(raw).trim() === '') return fallback;
|
||||
|
||||
const n = Number(raw);
|
||||
return Number.isFinite(n) && n >= 0 ? n : fallback;
|
||||
}
|
||||
|
||||
export function createState(env = process.env) {
|
||||
return {
|
||||
config: {
|
||||
kinds: parseKinds(env.SUBMISSION_KINDS, DEFAULTS.kinds),
|
||||
maxContentLength: parseNumber(env, 'SUBMISSION_MAX_CONTENT_LENGTH', DEFAULTS.maxContentLength),
|
||||
maxTags: parseNumber(env, 'SUBMISSION_MAX_TAGS', DEFAULTS.maxTags),
|
||||
maxPTags: parseNumber(env, 'SUBMISSION_MAX_P_TAGS', DEFAULTS.maxPTags),
|
||||
ipBurst: parseNumber(env, 'SUBMISSION_IP_BURST', DEFAULTS.ipBurst),
|
||||
ipRefillPerSec: parseNumber(env, 'SUBMISSION_IP_REFILL_PER_SEC', DEFAULTS.ipRefillPerSec),
|
||||
authorBurst: parseNumber(env, 'SUBMISSION_AUTHOR_BURST', DEFAULTS.authorBurst),
|
||||
authorRefillPerSec: parseNumber(env, 'SUBMISSION_AUTHOR_REFILL_PER_SEC', DEFAULTS.authorRefillPerSec),
|
||||
dedupeWindowMs: parseNumber(env, 'SUBMISSION_DEDUPE_WINDOW_MS', DEFAULTS.dedupeWindowMs),
|
||||
dedupeMinLength: parseNumber(env, 'SUBMISSION_DEDUPE_MIN_LENGTH', DEFAULTS.dedupeMinLength),
|
||||
},
|
||||
ipBuckets: new Map(),
|
||||
authorBuckets: new Map(),
|
||||
dedupe: new Map(),
|
||||
};
|
||||
}
|
||||
|
||||
// Lazily refills a token bucket and consumes one token. Returns true when a
|
||||
// token was available. `now` is in milliseconds.
|
||||
function takeToken(buckets, key, capacity, refillPerSec, now) {
|
||||
let bucket = buckets.get(key);
|
||||
|
||||
if (bucket === undefined) {
|
||||
bucket = { tokens: capacity, updated: now };
|
||||
buckets.set(key, bucket);
|
||||
} else {
|
||||
const elapsedSec = Math.max(0, now - bucket.updated) / 1000;
|
||||
bucket.tokens = Math.min(capacity, bucket.tokens + elapsedSec * refillPerSec);
|
||||
bucket.updated = now;
|
||||
}
|
||||
|
||||
if (bucket.tokens >= 1) {
|
||||
bucket.tokens -= 1;
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
// Drops buckets that have fully refilled to keep the maps bounded.
|
||||
function pruneBuckets(buckets, capacity) {
|
||||
if (buckets.size < 10000) return;
|
||||
|
||||
for (const [key, bucket] of buckets) {
|
||||
if (bucket.tokens >= capacity) buckets.delete(key);
|
||||
}
|
||||
}
|
||||
|
||||
function isDuplicate(dedupe, key, now, windowMs) {
|
||||
if (dedupe.size > 1000) {
|
||||
for (const [k, seenAt] of dedupe) {
|
||||
if (now - seenAt >= windowMs) dedupe.delete(k);
|
||||
}
|
||||
}
|
||||
|
||||
const seenAt = dedupe.get(key);
|
||||
return seenAt !== undefined && now - seenAt < windowMs;
|
||||
}
|
||||
|
||||
function reject(id, msg) {
|
||||
return { id, action: 'reject', msg };
|
||||
}
|
||||
|
||||
export function decide(msg, state, now = Date.now()) {
|
||||
const id = msg && msg.event && typeof msg.event.id === 'string' ? msg.event.id : '';
|
||||
|
||||
if (msg === null || typeof msg !== 'object') {
|
||||
return reject(id, 'malformed input');
|
||||
}
|
||||
|
||||
if (msg.type !== 'new') {
|
||||
return reject(id, `unexpected request type: ${String(msg.type)}`);
|
||||
}
|
||||
|
||||
const event = msg.event;
|
||||
if (event === null || typeof event !== 'object') {
|
||||
return reject(id, 'malformed event');
|
||||
}
|
||||
|
||||
const { kind, tags, content, pubkey } = event;
|
||||
|
||||
if (typeof pubkey !== 'string' || pubkey === '') {
|
||||
return reject(id, 'malformed event: missing pubkey');
|
||||
}
|
||||
if (!Number.isInteger(kind)) {
|
||||
return reject(id, 'malformed event: invalid kind');
|
||||
}
|
||||
if (!Array.isArray(tags)) {
|
||||
return reject(id, 'malformed event: invalid tags');
|
||||
}
|
||||
|
||||
const text = typeof content === 'string' ? content : '';
|
||||
const config = state.config;
|
||||
|
||||
if (!config.kinds.has(kind)) {
|
||||
return reject(id, `kind ${kind} not allowed`);
|
||||
}
|
||||
|
||||
if (Buffer.byteLength(text, 'utf8') > config.maxContentLength) {
|
||||
return reject(id, 'content too long');
|
||||
}
|
||||
|
||||
if (tags.length > config.maxTags) {
|
||||
return reject(id, 'too many tags');
|
||||
}
|
||||
|
||||
let pTags = 0;
|
||||
for (const tag of tags) {
|
||||
if (Array.isArray(tag) && tag[0] === 'p') pTags += 1;
|
||||
}
|
||||
if (pTags > config.maxPTags) {
|
||||
return reject(id, 'too many p tags (hellthread)');
|
||||
}
|
||||
|
||||
const source = typeof msg.sourceInfo === 'string' ? msg.sourceInfo : '';
|
||||
|
||||
pruneBuckets(state.ipBuckets, config.ipBurst);
|
||||
if (!takeToken(state.ipBuckets, source, config.ipBurst, config.ipRefillPerSec, now)) {
|
||||
return reject(id, 'rate limited');
|
||||
}
|
||||
|
||||
pruneBuckets(state.authorBuckets, config.authorBurst);
|
||||
if (!takeToken(state.authorBuckets, pubkey, config.authorBurst, config.authorRefillPerSec, now)) {
|
||||
return reject(id, 'rate limited');
|
||||
}
|
||||
|
||||
if (text.length >= config.dedupeMinLength) {
|
||||
const hash = createHash('sha256').update(text).digest('hex');
|
||||
if (isDuplicate(state.dedupe, hash, now, config.dedupeWindowMs)) {
|
||||
return reject(id, 'duplicate content');
|
||||
}
|
||||
state.dedupe.set(hash, now);
|
||||
}
|
||||
|
||||
return { id, action: 'accept', msg: '' };
|
||||
}
|
||||
|
||||
function writeResult(output, result) {
|
||||
output.write(`${JSON.stringify(result)}\n`);
|
||||
}
|
||||
|
||||
export async function run({ input = process.stdin, output = process.stdout, error = process.stderr, env = process.env, state = undefined } = {}) {
|
||||
const policyState = state ?? createState(env);
|
||||
const rl = readline.createInterface({ input, crlfDelay: Infinity });
|
||||
|
||||
for await (const line of rl) {
|
||||
if (line.trim() === '') continue;
|
||||
|
||||
let msg;
|
||||
try {
|
||||
msg = JSON.parse(line);
|
||||
} catch (e) {
|
||||
error.write(`[${NAME}] malformed JSON: ${e.message}\n`);
|
||||
writeResult(output, reject('', 'malformed JSON'));
|
||||
continue;
|
||||
}
|
||||
|
||||
let result;
|
||||
try {
|
||||
result = decide(msg, policyState);
|
||||
} catch (e) {
|
||||
error.write(`[${NAME}] unexpected error: ${e.stack || e.message}\n`);
|
||||
result = reject(msg && msg.event && typeof msg.event.id === 'string' ? msg.event.id : '', 'internal error');
|
||||
}
|
||||
|
||||
if (result.action !== 'accept') {
|
||||
error.write(`[${NAME}] reject id=${result.id} reason=${result.msg}\n`);
|
||||
}
|
||||
|
||||
writeResult(output, result);
|
||||
}
|
||||
}
|
||||
|
||||
const isMain = process.argv[1] !== undefined && import.meta.url === pathToFileURL(process.argv[1]).href;
|
||||
|
||||
if (isMain) {
|
||||
run().catch((e) => {
|
||||
process.stderr.write(`[${NAME}] fatal: ${e.stack || e.message}\n`);
|
||||
process.exit(1);
|
||||
});
|
||||
}
|
||||
Executable
+112
@@ -0,0 +1,112 @@
|
||||
#!/usr/bin/env node
|
||||
// Fail-closed strfry write policy for the members relay's approved-content sync.
|
||||
//
|
||||
// This policy only accepts events that arrived through `strfry sync` from the
|
||||
// dedicated staging relay. Everything else is rejected, including events with
|
||||
// the wrong source type or the wrong source URL, so that a misconfigured or
|
||||
// compromised sync source can never write into the production relay.
|
||||
//
|
||||
// Reads strfry JSONL input messages from stdin and writes one
|
||||
// {"id","action","msg"} JSON object per line to stdout.
|
||||
//
|
||||
// See README.md for the env vars and how strfry invokes this.
|
||||
|
||||
import readline from 'node:readline';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
|
||||
const NAME = 'sync-policy';
|
||||
|
||||
export function createState(env = process.env) {
|
||||
return {
|
||||
config: {
|
||||
stagingRelayUrl: typeof env.STAGING_RELAY_URL === 'string' ? env.STAGING_RELAY_URL.trim() : '',
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function reject(id, msg) {
|
||||
return { id, action: 'reject', msg };
|
||||
}
|
||||
|
||||
function describeSource(msg) {
|
||||
return `sourceType=${JSON.stringify(msg && msg.sourceType)} sourceInfo=${JSON.stringify(msg && msg.sourceInfo)}`;
|
||||
}
|
||||
|
||||
export function decide(msg, state) {
|
||||
const id = msg && msg.event && typeof msg.event.id === 'string' ? msg.event.id : '';
|
||||
const expected = state.config.stagingRelayUrl;
|
||||
|
||||
if (msg === null || typeof msg !== 'object') {
|
||||
return reject(id, 'malformed input');
|
||||
}
|
||||
|
||||
if (msg.type !== 'new') {
|
||||
return reject(id, `unexpected request type: ${String(msg.type)} (${describeSource(msg)})`);
|
||||
}
|
||||
|
||||
if (expected === '') {
|
||||
return reject(id, `source not allowed: STAGING_RELAY_URL is not configured (${describeSource(msg)})`);
|
||||
}
|
||||
|
||||
if (msg.sourceType !== 'Sync') {
|
||||
return reject(id, `source not allowed: expected sourceType=${JSON.stringify('Sync')} got ${JSON.stringify(msg.sourceType)} (${describeSource(msg)})`);
|
||||
}
|
||||
|
||||
if (msg.sourceInfo !== expected) {
|
||||
return reject(id, `source not allowed: expected sourceInfo=${JSON.stringify(expected)} got ${JSON.stringify(msg.sourceInfo)} (${describeSource(msg)})`);
|
||||
}
|
||||
|
||||
return { id, action: 'accept', msg: '' };
|
||||
}
|
||||
|
||||
function writeResult(output, result) {
|
||||
output.write(`${JSON.stringify(result)}\n`);
|
||||
}
|
||||
|
||||
export async function run({ input = process.stdin, output = process.stdout, error = process.stderr, env = process.env, state = undefined } = {}) {
|
||||
const policyState = state ?? createState(env);
|
||||
|
||||
if (policyState.config.stagingRelayUrl === '') {
|
||||
error.write(`[${NAME}] warning: STAGING_RELAY_URL is not set; rejecting all events\n`);
|
||||
} else {
|
||||
error.write(`[${NAME}] accepting sourceType=Sync sourceInfo=${JSON.stringify(policyState.config.stagingRelayUrl)}\n`);
|
||||
}
|
||||
|
||||
const rl = readline.createInterface({ input, crlfDelay: Infinity });
|
||||
|
||||
for await (const line of rl) {
|
||||
if (line.trim() === '') continue;
|
||||
|
||||
let msg;
|
||||
try {
|
||||
msg = JSON.parse(line);
|
||||
} catch (e) {
|
||||
error.write(`[${NAME}] malformed JSON: ${e.message}\n`);
|
||||
writeResult(output, reject('', 'malformed JSON'));
|
||||
continue;
|
||||
}
|
||||
|
||||
let result;
|
||||
try {
|
||||
result = decide(msg, policyState);
|
||||
} catch (e) {
|
||||
error.write(`[${NAME}] unexpected error: ${e.stack || e.message}\n`);
|
||||
result = reject(msg && msg.event && typeof msg.event.id === 'string' ? msg.event.id : '', 'internal error');
|
||||
}
|
||||
|
||||
if (result.action !== 'accept') {
|
||||
error.write(`[${NAME}] reject id=${result.id} reason=${result.msg}\n`);
|
||||
}
|
||||
|
||||
writeResult(output, result);
|
||||
}
|
||||
}
|
||||
|
||||
const isMain = process.argv[1] !== undefined && import.meta.url === pathToFileURL(process.argv[1]).href;
|
||||
|
||||
if (isMain) {
|
||||
run().catch((e) => {
|
||||
process.stderr.write(`[${NAME}] fatal: ${e.stack || e.message}\n`);
|
||||
process.exit(1);
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { Readable, Writable } from 'node:stream';
|
||||
|
||||
import { createState, decide, run } from '../submission-policy.mjs';
|
||||
|
||||
function newMsg(overrides = {}) {
|
||||
const { event: eventOverrides, ...rest } = overrides;
|
||||
|
||||
return {
|
||||
type: 'new',
|
||||
sourceType: 'IP4',
|
||||
sourceInfo: '10.0.0.1',
|
||||
event: {
|
||||
id: 'evt1',
|
||||
pubkey: 'pk1',
|
||||
kind: 360,
|
||||
tags: [],
|
||||
content: 'hello world',
|
||||
...(eventOverrides || {}),
|
||||
},
|
||||
...rest,
|
||||
};
|
||||
}
|
||||
|
||||
async function runLines(lines, { env, state } = {}) {
|
||||
const input = Readable.from(lines.map((line) => `${line}\n`));
|
||||
let stdout = '';
|
||||
let stderr = '';
|
||||
|
||||
const output = new Writable({ write(chunk, _enc, cb) { stdout += chunk.toString(); cb(); } });
|
||||
const error = new Writable({ write(chunk, _enc, cb) { stderr += chunk.toString(); cb(); } });
|
||||
|
||||
await run({ input, output, error, env, state });
|
||||
|
||||
const results = stdout.split('\n').filter((line) => line !== '').map((line) => JSON.parse(line));
|
||||
return { results, stdout, stderr };
|
||||
}
|
||||
|
||||
test('accepts a valid kind 360 submission', () => {
|
||||
const result = decide(newMsg(), createState({}));
|
||||
assert.deepEqual(result, { id: 'evt1', action: 'accept', msg: '' });
|
||||
});
|
||||
|
||||
test('accepts every kind in the default allowlist', () => {
|
||||
const state = createState({});
|
||||
|
||||
for (const kind of [0, 3, 5, 360, 30360]) {
|
||||
const result = decide(newMsg({ event: { id: `evt-${kind}`, kind } }), state, 1000);
|
||||
assert.equal(result.action, 'accept', `kind ${kind} should be accepted`);
|
||||
}
|
||||
});
|
||||
|
||||
test('rejects kinds outside the allowlist', () => {
|
||||
const state = createState({});
|
||||
|
||||
for (const kind of [1, 4, 7, 24133, 9735]) {
|
||||
const result = decide(newMsg({ event: { kind } }), state);
|
||||
assert.equal(result.action, 'reject');
|
||||
assert.match(result.msg, /not allowed/);
|
||||
}
|
||||
});
|
||||
|
||||
test('honours a custom SUBMISSION_KINDS allowlist', () => {
|
||||
const state = createState({ SUBMISSION_KINDS: '360,30360' });
|
||||
|
||||
assert.equal(decide(newMsg({ event: { kind: 360 } }), state).action, 'accept');
|
||||
assert.equal(decide(newMsg({ event: { kind: 30360 } }), state).action, 'accept');
|
||||
assert.equal(decide(newMsg({ event: { kind: 1 } }), state).action, 'reject');
|
||||
});
|
||||
|
||||
test('an empty allowlist denies everything', () => {
|
||||
const state = createState({ SUBMISSION_KINDS: '' });
|
||||
const result = decide(newMsg(), state);
|
||||
assert.equal(result.action, 'reject');
|
||||
});
|
||||
|
||||
test('rejects non-new message types', () => {
|
||||
const result = decide(newMsg({ type: 'lookback' }), createState({}));
|
||||
assert.equal(result.action, 'reject');
|
||||
assert.match(result.msg, /unexpected request type/);
|
||||
});
|
||||
|
||||
test('rejects malformed events', () => {
|
||||
const state = createState({});
|
||||
|
||||
assert.equal(decide(newMsg({ event: { pubkey: '' } }), state).action, 'reject');
|
||||
assert.equal(decide(newMsg({ event: { pubkey: 123 } }), state).action, 'reject');
|
||||
assert.equal(decide(newMsg({ event: { kind: 'x' } }), state).action, 'reject');
|
||||
assert.equal(decide(newMsg({ event: { tags: 'nope' } }), state).action, 'reject');
|
||||
assert.equal(decide(null, state).action, 'reject');
|
||||
assert.equal(decide(42, state).action, 'reject');
|
||||
});
|
||||
|
||||
test('rejects over-long content', () => {
|
||||
const state = createState({ SUBMISSION_MAX_CONTENT_LENGTH: '10' });
|
||||
|
||||
assert.equal(decide(newMsg({ event: { content: 'a'.repeat(10) } }), state).action, 'accept');
|
||||
assert.equal(decide(newMsg({ event: { content: 'a'.repeat(11) } }), state).action, 'reject');
|
||||
});
|
||||
|
||||
test('rejects events with too many tags', () => {
|
||||
const state = createState({ SUBMISSION_MAX_TAGS: '3' });
|
||||
const tags = [['e', '1'], ['e', '2'], ['e', '3'], ['e', '4']];
|
||||
|
||||
const result = decide(newMsg({ event: { tags } }), state);
|
||||
assert.equal(result.action, 'reject');
|
||||
assert.match(result.msg, /too many tags/);
|
||||
});
|
||||
|
||||
test('rejects hellthreads with too many p tags', () => {
|
||||
const state = createState({ SUBMISSION_MAX_P_TAGS: '2' });
|
||||
const tags = [['p', 'a'], ['p', 'b'], ['p', 'c']];
|
||||
|
||||
const result = decide(newMsg({ event: { tags } }), state);
|
||||
assert.equal(result.action, 'reject');
|
||||
assert.match(result.msg, /hellthread/);
|
||||
});
|
||||
|
||||
test('enforces the per-IP token bucket and refills over time', () => {
|
||||
const state = createState({ SUBMISSION_IP_BURST: '2', SUBMISSION_IP_REFILL_PER_SEC: '1' });
|
||||
|
||||
assert.equal(decide(newMsg({ event: { id: 'a', pubkey: 'pk1', content: 'a' } }), state, 1000).action, 'accept');
|
||||
assert.equal(decide(newMsg({ event: { id: 'b', pubkey: 'pk2', content: 'b' } }), state, 1000).action, 'accept');
|
||||
|
||||
const limited = decide(newMsg({ event: { id: 'c', pubkey: 'pk3', content: 'c' } }), state, 1000);
|
||||
assert.equal(limited.action, 'reject');
|
||||
assert.match(limited.msg, /rate limited/);
|
||||
|
||||
assert.equal(decide(newMsg({ event: { id: 'd', pubkey: 'pk4', content: 'd' } }), state, 2000).action, 'accept');
|
||||
});
|
||||
|
||||
test('enforces the per-author token bucket and refills over time', () => {
|
||||
const state = createState({ SUBMISSION_AUTHOR_BURST: '2', SUBMISSION_AUTHOR_REFILL_PER_SEC: '1' });
|
||||
|
||||
assert.equal(decide(newMsg({ event: { id: 'a', content: 'a' } }), state, 1000).action, 'accept');
|
||||
assert.equal(decide(newMsg({ event: { id: 'b', content: 'b' } }), state, 1000).action, 'accept');
|
||||
|
||||
const limited = decide(newMsg({ event: { id: 'c', content: 'c' } }), state, 1000);
|
||||
assert.equal(limited.action, 'reject');
|
||||
assert.match(limited.msg, /rate limited/);
|
||||
|
||||
assert.equal(decide(newMsg({ event: { id: 'd', content: 'd' } }), state, 2000).action, 'accept');
|
||||
});
|
||||
|
||||
test('dedupes identical long content within the window', () => {
|
||||
const state = createState({});
|
||||
const content = 'x'.repeat(60);
|
||||
|
||||
assert.equal(decide(newMsg({ event: { id: 'a', content } }), state, 1000).action, 'accept');
|
||||
|
||||
const duplicate = decide(newMsg({ event: { id: 'b', content } }), state, 1000);
|
||||
assert.equal(duplicate.action, 'reject');
|
||||
assert.match(duplicate.msg, /duplicate content/);
|
||||
|
||||
assert.equal(decide(newMsg({ event: { id: 'c', content } }), state, 61001).action, 'accept');
|
||||
});
|
||||
|
||||
test('does not dedupe short content', () => {
|
||||
const state = createState({});
|
||||
|
||||
assert.equal(decide(newMsg({ event: { id: 'a', content: 'hi' } }), state, 1000).action, 'accept');
|
||||
assert.equal(decide(newMsg({ event: { id: 'b', content: 'hi' } }), state, 1000).action, 'accept');
|
||||
});
|
||||
|
||||
test('run() processes JSONL end-to-end', async () => {
|
||||
const { results } = await runLines([
|
||||
JSON.stringify(newMsg({ event: { id: 'a', kind: 360 } })),
|
||||
JSON.stringify(newMsg({ event: { id: 'b', kind: 1 } })),
|
||||
]);
|
||||
|
||||
assert.deepEqual(results, [
|
||||
{ id: 'a', action: 'accept', msg: '' },
|
||||
{ id: 'b', action: 'reject', msg: 'kind 1 not allowed' },
|
||||
]);
|
||||
});
|
||||
|
||||
test('run() rejects malformed JSON without crashing and keeps going', async () => {
|
||||
const { results, stderr } = await runLines([
|
||||
'{not json',
|
||||
JSON.stringify(newMsg({ event: { id: 'ok' } })),
|
||||
'also not json',
|
||||
]);
|
||||
|
||||
assert.equal(results.length, 3);
|
||||
assert.equal(results[0].action, 'reject');
|
||||
assert.equal(results[1].action, 'accept');
|
||||
assert.equal(results[2].action, 'reject');
|
||||
assert.match(stderr, /malformed JSON/);
|
||||
});
|
||||
|
||||
test('run() skips blank lines', async () => {
|
||||
const { results } = await runLines(['', ' ', JSON.stringify(newMsg({ event: { id: 'ok' } }))]);
|
||||
assert.equal(results.length, 1);
|
||||
assert.equal(results[0].id, 'ok');
|
||||
});
|
||||
|
||||
test('run() turns unexpected errors into rejections', async () => {
|
||||
const state = { config: { kinds: { has() { throw new Error('boom'); } } } };
|
||||
|
||||
const { results, stderr } = await runLines([JSON.stringify(newMsg({ event: { id: 'boom' } }))], { state });
|
||||
|
||||
assert.equal(results.length, 1);
|
||||
assert.deepEqual(results[0], { id: 'boom', action: 'reject', msg: 'internal error' });
|
||||
assert.match(stderr, /unexpected error/);
|
||||
});
|
||||
@@ -0,0 +1,135 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { Readable, Writable } from 'node:stream';
|
||||
|
||||
import { createState, decide, run } from '../sync-policy.mjs';
|
||||
|
||||
const STAGING = 'ws://staging-relay:7779';
|
||||
|
||||
function syncMsg(overrides = {}) {
|
||||
const { event: eventOverrides, ...rest } = overrides;
|
||||
|
||||
return {
|
||||
type: 'new',
|
||||
sourceType: 'Sync',
|
||||
sourceInfo: STAGING,
|
||||
event: {
|
||||
id: 'evt1',
|
||||
pubkey: 'pk1',
|
||||
kind: 360,
|
||||
tags: [],
|
||||
content: 'hello world',
|
||||
...(eventOverrides || {}),
|
||||
},
|
||||
...rest,
|
||||
};
|
||||
}
|
||||
|
||||
async function runLines(lines, { env, state } = {}) {
|
||||
const input = Readable.from(lines.map((line) => `${line}\n`));
|
||||
let stdout = '';
|
||||
let stderr = '';
|
||||
|
||||
const output = new Writable({ write(chunk, _enc, cb) { stdout += chunk.toString(); cb(); } });
|
||||
const error = new Writable({ write(chunk, _enc, cb) { stderr += chunk.toString(); cb(); } });
|
||||
|
||||
await run({ input, output, error, env, state });
|
||||
|
||||
const results = stdout.split('\n').filter((line) => line !== '').map((line) => JSON.parse(line));
|
||||
return { results, stdout, stderr };
|
||||
}
|
||||
|
||||
test('accepts Sync events from the configured staging relay', () => {
|
||||
const state = createState({ STAGING_RELAY_URL: STAGING });
|
||||
assert.deepEqual(decide(syncMsg(), state), { id: 'evt1', action: 'accept', msg: '' });
|
||||
});
|
||||
|
||||
test('accepts any kind from the configured staging relay', () => {
|
||||
const state = createState({ STAGING_RELAY_URL: STAGING });
|
||||
|
||||
for (const kind of [0, 1, 3, 7, 360, 30360, 24133, 9735]) {
|
||||
const result = decide(syncMsg({ event: { id: `evt-${kind}`, kind } }), state);
|
||||
assert.equal(result.action, 'accept', `kind ${kind} should be accepted`);
|
||||
}
|
||||
});
|
||||
|
||||
test('rejects events that did not arrive via Sync', () => {
|
||||
const state = createState({ STAGING_RELAY_URL: STAGING });
|
||||
|
||||
for (const sourceType of ['IP4', 'IP6', 'Import', 'Stream', 'Stored']) {
|
||||
const result = decide(syncMsg({ sourceType }), state);
|
||||
assert.equal(result.action, 'reject', `${sourceType} should be rejected`);
|
||||
assert.match(result.msg, /expected sourceType/);
|
||||
}
|
||||
});
|
||||
|
||||
test('rejects Sync events from the wrong source URL', () => {
|
||||
const state = createState({ STAGING_RELAY_URL: STAGING });
|
||||
const result = decide(syncMsg({ sourceInfo: 'ws://evil.example:7779' }), state);
|
||||
|
||||
assert.equal(result.action, 'reject');
|
||||
assert.match(result.msg, /expected sourceInfo/);
|
||||
});
|
||||
|
||||
test('compares sourceInfo byte-exactly', () => {
|
||||
const state = createState({ STAGING_RELAY_URL: STAGING });
|
||||
|
||||
assert.equal(decide(syncMsg({ sourceInfo: `${STAGING} ` }), state).action, 'reject');
|
||||
assert.equal(decide(syncMsg({ sourceInfo: `${STAGING}/` }), state).action, 'reject');
|
||||
assert.equal(decide(syncMsg({ sourceInfo: STAGING.toUpperCase() }), state).action, 'reject');
|
||||
});
|
||||
|
||||
test('trims whitespace around the configured STAGING_RELAY_URL', () => {
|
||||
const state = createState({ STAGING_RELAY_URL: ` ${STAGING}\n` });
|
||||
assert.equal(decide(syncMsg(), state).action, 'accept');
|
||||
});
|
||||
|
||||
test('fails closed when STAGING_RELAY_URL is not configured', () => {
|
||||
const state = createState({});
|
||||
const result = decide(syncMsg(), state);
|
||||
|
||||
assert.equal(result.action, 'reject');
|
||||
assert.match(result.msg, /not configured/);
|
||||
});
|
||||
|
||||
test('rejects non-new message types', () => {
|
||||
const state = createState({ STAGING_RELAY_URL: STAGING });
|
||||
const result = decide(syncMsg({ type: 'lookback' }), state);
|
||||
|
||||
assert.equal(result.action, 'reject');
|
||||
assert.match(result.msg, /unexpected request type/);
|
||||
});
|
||||
|
||||
test('rejects malformed input', () => {
|
||||
const state = createState({ STAGING_RELAY_URL: STAGING });
|
||||
|
||||
assert.equal(decide(null, state).action, 'reject');
|
||||
assert.match(decide(null, state).msg, /malformed input/);
|
||||
});
|
||||
|
||||
test('run() accepts an expected event and rejects the rest', async () => {
|
||||
const { results, stderr } = await runLines(
|
||||
[
|
||||
JSON.stringify(syncMsg({ event: { id: 'a' } })),
|
||||
JSON.stringify(syncMsg({ event: { id: 'b' }, sourceType: 'IP4' })),
|
||||
JSON.stringify(syncMsg({ event: { id: 'c' }, sourceInfo: 'ws://evil.example:7779' })),
|
||||
],
|
||||
{ env: { STAGING_RELAY_URL: STAGING } },
|
||||
);
|
||||
|
||||
assert.deepEqual(results.map((r) => r.action), ['accept', 'reject', 'reject']);
|
||||
assert.match(stderr, /expected sourceType/);
|
||||
assert.match(stderr, /expected sourceInfo/);
|
||||
});
|
||||
|
||||
test('run() rejects malformed JSON without crashing and keeps going', async () => {
|
||||
const { results, stderr } = await runLines(
|
||||
['{not json', JSON.stringify(syncMsg({ event: { id: 'ok' } }))],
|
||||
{ env: { STAGING_RELAY_URL: STAGING } },
|
||||
);
|
||||
|
||||
assert.equal(results.length, 2);
|
||||
assert.equal(results[0].action, 'reject');
|
||||
assert.equal(results[1].action, 'accept');
|
||||
assert.match(stderr, /malformed JSON/);
|
||||
});
|
||||
Reference in new issue
Block a user