> ## Documentation Index > Fetch the complete documentation index at: https://partners.docs.polymarket.us/llms.txt > Use this file to discover all available pages before exploring further. # Reconnect and recover gaps > The subscription session model, the revoked-session error and its workaround, the real timeouts, and per-stream gap recovery. Every gRPC stream you open will close, and what you do in the next second decides whether you lose events or double-process them. Before this page: a working subscription on at least one stream — [Drop copy](/streams#drop-copy), [Position change](/streams#position-change) — and the caps from [Choose a stream](/streams#choose-a-stream). **API reference:** Stream error handling. Opens on the public documentation site in a new tab. Where it disagrees with this page, this page is authoritative for the partner surface. ## The session model **A subscription session is keyed on caller identity: the token subject plus the participant header.** It is **not** keyed on the accounts or the symbols you requested. That single fact explains the class of failure partners spend weeks on. Two processes using the same client credentials and the same `x-participant-id` are, to the subscription manager, the same session — not two sessions that happen to want different symbols. Nine partners independently investigated this. Give every long-lived subscriber its own distinguishable caller identity where you can, and do not assume that requesting different symbols isolates two subscribers from each other. ## `13 INTERNAL: Subscription manager revoked session` ``` 13 INTERNAL: Subscription manager revoked session ``` **Known workaround: pass an explicit list of markets on `CreateOrderSubscription`.** There is no symbol cap on that path, so naming your markets explicitly does not trade one limit for another. This has been shared one partner at a time; it is not in the published docs. **Not yet published.** What is confirmed is the keying — token subject plus participant header — and the explicit-market-list workaround. Anything beyond that, including whether a second subscription under the same caller identity always revokes the first, is unconfirmed. Ask your integration lead before you design around a specific trigger. Treat `13 INTERNAL` as **terminal for that session and immediately retryable as a new subscribe**. Do not treat it as a transport blip to be retried on the same call object. ## The timeouts that actually apply | Boundary | Value | Scope | How it surfaces | | ------------------------ | ------------------------------- | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | API Gateway idle timeout | 30 seconds | Per request at the edge | **gRPC streams bypass it.** A quiet stream is not killed at 30 seconds | | ALB timeout | **10 minutes** | Per connection | `SocketError: other side closed` | | Edge request timeout | 30 seconds | Per request at the edge | A **504** means the request took over 30 seconds at the edge. `POST /v1/report/trades/search` unfiltered hits it about two thirds of the time; the same call with a symbol filter returns in \~130 ms, reliably | | Access-token lifetime | `expires_in` is **180** seconds | Per token | Honour `expires_in` minus a 30-second buffer. **Never hardcode 180** — four current pages do | Whether a stream that was authorized with a token outlives that token's expiry is not published. Until it is confirmed, keep refreshing on the `expires_in` minus 30-second schedule and be ready for a close at any point — the reconnect wrapper below mints a fresh token on every attempt, which is correct either way. ## Reconnect cadence and backoff A 10-minute ALB close is expected operation, not a fault. Re-subscribe at once with a freshly minted token. From 1 second, doubling to a 30-second ceiling, with full jitter. Keep the ceiling under the 10-minute connection life so a recovering stream has time to do useful work. `StreamRFQEvents` is capped at **1 open/sec per firm** — the only published open-rate number. Hold every stream to that rate so a reconnect storm cannot become a rate-limit incident. You have **20 concurrent streams per firm** across all gRPC subscriptions. A reconnect loop that opens before the old stream is fully closed can spend budget you have already allocated. `PERMISSION_DENIED` (missing scope grant), `Aborted` (a suppressed ledger entry type) and `InvalidArgument` will not fix themselves. Retrying them burns budget and hides the defect. **Not yet published.** No reconnect budget is published, and whether opening a stream draws on the per-firm unary budget (**250 requests / 60 s** in cluster configuration) is an open gap. The cadence above is client-side guidance chosen to stay inside the numbers that *are* published — confirm it before you run hundreds of subscribers. ## Gap recovery, per stream | Stream | On reconnect | What you must do | | ----------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | Drop copy | Resume exactly — `resume_token` is populated, \~576 bytes | Pass the last persisted token. Dedupe on `clord_id` plus the execution identifier | | Position change | Resume — `resume_token` is populated | Pass the last persisted token, then reconcile against your own position table | | Instrument state change | **No resume.** It declares `resume_token` and never populates it | Re-read instrument state for your working set. `ListInstruments` and `ListSymbols` are **6/min per firm** each, so re-read a tracked set, not the universe | | Balance ledger | Replay phase runs again | Dedupe on the entry identifier; the replay *is* the recovery | | Market data | **No resume field at all** | Take the snapshot again. Subscribe is snapshot-then-delta, so a fresh subscribe gives you current state | Two rules cover all five: **Snapshot then delta.** Subscribe delivers current state first and changes after. Do not apply deltas to a state you built before the disconnect. **At-least-once, so dedupe.** Redelivery is expected on every stream. Every handler must be idempotent, keyed on the event's own identifier, never on arrival order. ## Sharding past 1000 instruments Work the arithmetic before you write the subscriber. ``` 1,000 instruments per market-data stream (cap is per stream) 20 concurrent streams per firm (cap is per firm, ALL subscription types) ------------------------------------------------------------------ 20,000 instruments — theoretical ceiling, using every stream for market data ``` You do not get all 20. A realistic ISV allocation: ``` 1 drop copy 1 position change 1 balance ledger (per account — see /streams#balance-ledger) --- 3 reserved 17 available for market data × 1,000 = 17,000 instruments ``` **Above roughly 17,000 instruments, this does not fit, and there is no published sharding pattern for it.** Say so to your integration lead rather than working around it: one partner ran 7,372 subscriptions before being told the cap existed and had to re-architect in production. Do not reach for an empty `symbols` list. That subscribes to **all** instruments and collides with the same 1000-per-stream cap. What you can do today, inside the caps: Assign each symbol to bucket `hash(symbol) mod N` with N ≤ 17 and ≤1000 symbols per bucket. Fixed buckets mean a reconnect re-subscribes the same symbol set, so your dedupe keys stay stable. Market data streams are the only cap-bound-by-count subscription. Drive the tracked set from open positions and live orders, and drop symbols you have no exposure to. Position change is position-driven and needs no symbol list at all, so exposure monitoring costs you one stream regardless of instrument count. ## A reconnect wrapper Mint a fresh token per attempt, persist the resume token as you go, and let non-transient statuses out. ```python Python theme={null} import random import time import grpc from polymarket.v1 import dropcopy_pb2, dropcopy_pb2_grpc # generated from polymarket-protos.zip from polymarket_auth import PolymarketClient # canonical client: /connect#authentication # The gRPC target is GRPC_TARGET in polymarket_auth.py — one place, both environments. pmx = PolymarketClient() PARTICIPANT_ID = "firms/20260821-examplefirminc-api-participant/participants/p-0f31c9" # participantId from kyc.approved, verbatim ACCOUNT = "PASTE_PROVISIONED_ACCOUNT" # provisionedAccount from kyc.approved, verbatim RESUME_FILE = "./dropcopy.resume" BACKOFF_MIN = 1.0 # seconds; also the floor set by the 1 open/sec rate on StreamRFQEvents BACKOFF_MAX = 30.0 # seconds; under the 10-minute ALB connection life FATAL = { grpc.StatusCode.PERMISSION_DENIED, # missing scope grant — re-mint after we add it grpc.StatusCode.ABORTED, # suppressed ledger entry type requested grpc.StatusCode.INVALID_ARGUMENT, # request defect } def subscribe_once() -> None: """One connection. Returns when the stream ends; raises on a fatal status.""" with pmx.channel() as channel: stub = dropcopy_pb2_grpc.DropCopyAPIStub(channel) request = dropcopy_pb2.CreateDropCopySubscriptionRequest(accounts=[ACCOUNT]) try: with open(RESUME_FILE, "rb") as fh: token = fh.read() if token: request.resume_token = token except FileNotFoundError: pass # no token yet: fresh subscription, not a replay stream = stub.CreateDropCopySubscription( request, # Built per attempt, so each reconnect carries a live token. metadata=pmx.metadata(participant_id=PARTICIPANT_ID), ) for message in stream: handle(message) # must be idempotent: delivery is at-least-once if message.resume_token: with open(RESUME_FILE, "wb") as fh: fh.write(message.resume_token) def handle(message) -> None: print(message) def run_forever() -> None: backoff = BACKOFF_MIN while True: try: subscribe_once() backoff = BACKOFF_MIN # clean end: the ALB 10-minute close lands here print("stream closed; reconnecting immediately") time.sleep(BACKOFF_MIN) continue except grpc.RpcError as err: code, details = err.code(), err.details() if code in FATAL: raise RuntimeError(f"not retryable: {code.name}: {details}") from err # 13 INTERNAL "Subscription manager revoked session" is terminal for the session and # retryable as a NEW subscribe. SocketError / UNAVAILABLE is the 10-minute ALB close. print(f"reconnecting after {code.name}: {details}") time.sleep(backoff * random.random()) # full jitter backoff = min(backoff * 2, BACKOFF_MAX) if __name__ == "__main__": run_forever() ``` ```typescript TypeScript theme={null} import { readFileSync, writeFileSync } from "node:fs"; import * as grpc from "@grpc/grpc-js"; import * as protoLoader from "@grpc/proto-loader"; import { PolymarketClient } from "./polymarketAuth"; // canonical client: /connect#authentication const pmx = new PolymarketClient(); const GRPC_TARGET = "grpc-api.preprod.polymarketexchange.com:443"; const PARTICIPANT_ID = "firms/20260821-examplefirminc-api-participant/participants/p-0f31c9"; // participantId from kyc.approved, verbatim const ACCOUNT = "PASTE_PROVISIONED_ACCOUNT"; // provisionedAccount from kyc.approved, verbatim const RESUME_FILE = "./dropcopy.resume"; const PROTO = "./protos/polymarket/v1/dropcopy.proto"; // from polymarket-protos.zip const BACKOFF_MIN_MS = 1_000; // also the floor set by the 1 open/sec rate on StreamRFQEvents const BACKOFF_MAX_MS = 30_000; // under the 10-minute ALB connection life const FATAL = new Set([ grpc.status.PERMISSION_DENIED, // missing scope grant — re-mint after we add it grpc.status.ABORTED, // suppressed ledger entry type requested grpc.status.INVALID_ARGUMENT, // request defect ]); const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms)); function subscribeOnce(): Promise { return new Promise(async (resolve, reject) => { const pkg = grpc.loadPackageDefinition( protoLoader.loadSync(PROTO, { keepCase: true, defaults: true, includeDirs: ["./protos"] }), ) as any; const client = new pkg.polymarket.v1.DropCopyAPI(GRPC_TARGET, grpc.credentials.createSsl()); // Built per attempt, so each reconnect carries a live token. const metadata = await pmx.metadata(PARTICIPANT_ID); const request: Record = { accounts: [ACCOUNT] }; try { const token = readFileSync(RESUME_FILE); if (token.length > 0) request.resume_token = token; } catch { /* no token yet: fresh subscription, not a replay */ } const stream = client.CreateDropCopySubscription(request, metadata); stream.on("data", (message: any) => { handle(message); // must be idempotent: delivery is at-least-once if (message.resume_token?.length) writeFileSync(RESUME_FILE, message.resume_token); }); stream.on("error", (err: grpc.ServiceError) => reject(err)); stream.on("end", resolve); }); } function handle(message: unknown): void { console.log(message); } export async function runForever(): Promise { let backoff = BACKOFF_MIN_MS; for (;;) { try { await subscribeOnce(); backoff = BACKOFF_MIN_MS; // clean end: the ALB 10-minute close lands here console.log("stream closed; reconnecting immediately"); await sleep(BACKOFF_MIN_MS); continue; } catch (err) { const e = err as grpc.ServiceError; if (FATAL.has(e.code)) throw new Error(`not retryable: ${grpc.status[e.code]}: ${e.details}`); // 13 INTERNAL "Subscription manager revoked session" is terminal for the session and retryable // as a NEW subscribe. UNAVAILABLE / socket close is the 10-minute ALB close. console.log(`reconnecting after ${grpc.status[e.code]}: ${e.details}`); } await sleep(Math.random() * backoff); // full jitter backoff = Math.min(backoff * 2, BACKOFF_MAX_MS); } } runForever().catch((err) => { console.error(err); process.exit(1); }); ``` ```go Go theme={null} package main import ( "context" "fmt" "log" "math/rand" "os" "time" "google.golang.org/grpc" "google.golang.org/grpc/codes" "google.golang.org/grpc/credentials" "google.golang.org/grpc/metadata" "google.golang.org/grpc/status" pmauth "example.com/yourfirm/pmauth" // canonical client: /connect#authentication dc "github.com/your-firm/gen/polymarket/v1" // generated from polymarket-protos.zip ) const ( grpcTarget = "grpc-api.preprod.polymarketexchange.com:443" participantID = "firms/20260821-examplefirminc-api-participant/participants/p-0f31c9" // from kyc.approved, verbatim account = "PASTE_PROVISIONED_ACCOUNT" // provisionedAccount from kyc.approved, verbatim resumeFile = "./dropcopy.resume" backoffMin = 1 * time.Second // also the floor set by the 1 open/sec rate on StreamRFQEvents backoffMax = 30 * time.Second // under the 10-minute ALB connection life ) // Missing scope grant, suppressed ledger entry type, request defect: none of these fix themselves. var fatal = map[codes.Code]bool{ codes.PermissionDenied: true, codes.Aborted: true, codes.InvalidArgument: true, } // One auth client for the life of the process: it caches the token and re-mints // at expires_in minus 30 s. var pm = pmauth.New() func subscribeOnce(ctx context.Context) error { // Built per attempt, so each reconnect carries a live token. callMD, err := pm.Metadata(participantID) if err != nil { return err } conn, err := grpc.NewClient(grpcTarget, grpc.WithTransportCredentials( credentials.NewClientTLSFromCert(nil, ""), )) if err != nil { return err } defer conn.Close() ctx = metadata.NewOutgoingContext(ctx, callMD) req := &dc.CreateDropCopySubscriptionRequest{Accounts: []string{account}} if resume, err := os.ReadFile(resumeFile); err == nil && len(resume) > 0 { req.ResumeToken = resume // no token yet means a fresh subscription, not a replay } stream, err := dc.NewDropCopyAPIClient(conn).CreateDropCopySubscription(ctx, req) if err != nil { return err } for { message, err := stream.Recv() if err != nil { return err } fmt.Println(message.String()) // handler must be idempotent: delivery is at-least-once if t := message.GetResumeToken(); len(t) > 0 { if err := os.WriteFile(resumeFile, t, 0o600); err != nil { return err } } } } func main() { backoff := backoffMin for { err := subscribeOnce(context.Background()) code := status.Code(err) if fatal[code] { log.Fatalf("not retryable: %s: %s", code, status.Convert(err).Message()) } // 13 INTERNAL "Subscription manager revoked session" is terminal for the session and // retryable as a NEW subscribe. Unavailable is the 10-minute ALB close. log.Printf("reconnecting after %s: %s", code, status.Convert(err).Message()) time.Sleep(time.Duration(rand.Int63n(int64(backoff)))) // full jitter if backoff *= 2; backoff > backoffMax { backoff = backoffMax } } } ``` ## What can go wrong | Symptom | Cause | What you do | | --------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | `13 INTERNAL: Subscription manager revoked session` | The session is keyed on caller identity — token subject plus participant header — not on your requested accounts or symbols | Re-subscribe as a new session. Pass an explicit list of markets on `CreateOrderSubscription`, which has no symbol cap on that path | | `SocketError: other side closed` every \~10 minutes | ALB timeout, 10 minutes per connection. Expected operation | Reconnect immediately with the persisted `resume_token` | | `504` | The request took over 30 seconds at the edge | Filter the request. `POST /v1/report/trades/search` unfiltered 504s about two thirds of the time; with a symbol filter it returns in \~130 ms | | A quiet stream dies at 30 seconds | Not the API Gateway idle timeout — gRPC streams bypass it | Look for the ALB close, a revoked session, or your own client-side deadline | | Events missing after a reconnect | You reconnected on a stream with no populated `resume_token` | Instrument state change and market data recover by reading, not resuming | | Duplicated payouts or double-counted fills | Delivery is at-least-once and you keyed on arrival order | Dedupe on the event's own identifier. Never pay a user off a stream event alone — see [Who credits the user](/settlement#who-credits-the-user) | | Reconnect loop hits `PERMISSION_DENIED` forever | A scope grant is missing, or the token predates the grant | Stop retrying. Scopes are granted server-side; re-mint after we add the grant | | Pre-maintenance execution queries return empty | Stored execution history is archived during maintenance | Persist fills from drop copy yourself; do not plan to backfill from query endpoints | ## Next [Settlement](/settlement)