Coverage for server / utilities / refresh_single_flight.py: 94%
78 statements
« prev ^ index » next coverage.py v7.13.4, created at 2026-10-04 09:33 +0000
1"""
2Single-flight refresh: one rotation per refresh token, even under a race.
4WHAT THIS FIXES (Zephyr EI-T430). Two tabs — or one tab on a flaky network —
5can fire ``POST /v1/auth/refresh`` with the SAME refresh-token cookie at the
6same moment. Both reach Auth0, Auth0's reuse leeway lets both succeed, and the
7caller ends up holding TWO independently valid access tokens. Our access tokens
8carry no ``jti`` and no ``sid`` and live for 24 hours, so that second session
9cannot be revoked and does not quietly expire: it is a full day of parallel
10access nobody intended to issue.
12IT WORKS WITH AUTH0 RATHER THAN AGAINST IT. Nothing here disables rotation,
13changes tenant configuration, or calls Auth0 differently. It calls Auth0
14*exactly once* per refresh token instead of twice, which is strictly friendlier
15to the rotation model than what we do today — today both racers spend the same
16token and rely on the leeway to forgive it. The leeway stays exactly where it
17is, as Auth0's safety net. We simply stop needing it.
19HOW. The first request to arrive claims the token atomically (SET NX) and does
20the real exchange. Any request that finds the claim already taken waits briefly
21for the winner's result and adopts THAT session — same access token, same
22refresh token, same CSRF token — instead of minting a second one. One refresh
23token in, one session out.
25IT FAILS OPEN, DELIBERATELY. If Redis is unreachable, ``claim`` reports success
26so the caller proceeds exactly as it does today, and Auth0's leeway absorbs the
27race as it always has. An auth path must not go down because a cache did; the
28worst case is that we are no better than before, never worse.
30WHAT IT DOES NOT DO. It prevents a duplicate session from being CREATED. It
31cannot retire an access token already minted by an earlier race, because that
32token is a stateless JWT with no identifier to deny by. Closing that gap needs a
33per-request denylist keyed on a token fingerprint — a Redis round-trip on every
34authenticated request — which is a much larger change and is deliberately not
35attempted here.
37ON STORING TOKENS. The winner's tokens sit in Redis only long enough for a
38racing sibling to collect them (``_RESULT_TTL_SECONDS``), keyed by a SHA-256
39fingerprint of the refresh token — never by the token itself. This is the same
40material the caller already holds in its cookies, and the window is seconds.
42Developer: Allan Ninal
43Date: 2026-09-21
44"""
46import asyncio
47import hashlib
48import json
49import logging
50from typing import Any, Optional
52import redis.asyncio as aioredis
54from server.utilities.redis_url import redis_url_or_default
56logger = logging.getLogger(__name__)
58# THE SAFETY CEILING. Auth0's refresh rotation forgives a token re-spent within
59# this window; past it, reuse detection rejects the token and kills the family.
60# Everything below MUST stay inside it.
61#
62# This is not a tuning knob. A result that outlives Auth0's leeway is a session
63# handed out for a token Auth0 would have REFUSED, minted from our cache without
64# ever asking Auth0 — i.e. we would be more permissive than the authority we are
65# supposed to be deferring to, and a replayed refresh token would buy a live
66# session for as long as our window lasted.
67#
68# Caught by tests/account/test_zephyr_edge_refresh_race.py's sibling
69# EI-T428 (teacher-student-automation), which re-spends a consumed token after
70# 5s and requires a 401: an earlier revision of this file used a 10s result TTL
71# and served that replay a valid session.
72_AUTH0_REUSE_LEEWAY_SECONDS = 3
74# How long the winner's result stays collectable. STRICTLY inside the leeway, so
75# we can never answer where Auth0 would have refused. A genuine racer is already
76# in flight and collects this within milliseconds.
77_RESULT_TTL_SECONDS = 2
79# How long a claim is held. This one only makes a racer WAIT — it never serves a
80# session — so it is not the security-critical value, but it is kept at the
81# leeway so a crashed winner cannot stall a real retry.
82_CLAIM_TTL_SECONDS = _AUTH0_REUSE_LEEWAY_SECONDS
84# How long a loser waits before giving up and asking Auth0 itself.
85#
86# NOT a security value, unlike the two above. It cannot widen the replay window
87# — only _RESULT_TTL_SECONDS can, and that is measured from the moment Auth0
88# consumed the token, the same instant Auth0's own leeway starts counting. This
89# one only decides how patient a racer is before doing the work itself.
90#
91# It must EXCEED a real Auth0 round trip, because the winner publishes only
92# after Auth0 answers. A racer that gives up first falls through and mints the
93# second session this module exists to prevent — the invariant degrades
94# silently, under load, exactly when it matters.
95#
96# Measured against QA on 2026-09-21 over four sequential exchanges:
97# 900ms, 900ms, 1522ms, 946ms. An earlier revision used 1.5s, which the slowest
98# of those already beat: no headroom at all.
99_AUTH0_EXCHANGE_BUDGET_SECONDS = 1.6
100_WAIT_TIMEOUT_SECONDS = 2.5
101_POLL_INTERVAL_SECONDS = 0.05
103if _RESULT_TTL_SECONDS >= _AUTH0_REUSE_LEEWAY_SECONDS: # pragma: no cover
104 raise AssertionError(
105 "refresh_single_flight would outlive Auth0's reuse leeway: a replayed "
106 "refresh token could be served a session from cache that Auth0 itself "
107 "would have refused."
108 )
110if _WAIT_TIMEOUT_SECONDS <= _AUTH0_EXCHANGE_BUDGET_SECONDS: # pragma: no cover
111 raise AssertionError(
112 "refresh_single_flight would give up before a normal Auth0 exchange "
113 "completes, so a racer would mint the second session this module exists "
114 "to prevent."
115 )
117_KEY_PREFIX = "auth:refresh"
119_client: Optional[aioredis.Redis] = None
122def _fingerprint(refresh_token: str) -> str:
123 """A stable key for a token, without putting the token in the keyspace."""
124 return hashlib.sha256(refresh_token.encode("utf-8")).hexdigest()
127def _claim_key(refresh_token: str) -> str:
128 return f"{_KEY_PREFIX}:claim:{_fingerprint(refresh_token)}"
131def _result_key(refresh_token: str) -> str:
132 return f"{_KEY_PREFIX}:result:{_fingerprint(refresh_token)}"
135async def _get_client() -> Optional[aioredis.Redis]:
136 """The shared client, or None if Redis cannot be reached."""
137 global _client
138 if _client is not None:
139 return _client
140 try:
141 _client = aioredis.from_url(
142 redis_url_or_default("redis://localhost:6379/0"),
143 decode_responses=True,
144 socket_connect_timeout=2,
145 socket_timeout=2,
146 )
147 return _client
148 except Exception:
149 logger.warning(
150 "Single-flight refresh: Redis unavailable; racing refreshes will be "
151 "absorbed by Auth0's reuse leeway as before.",
152 exc_info=True,
153 )
154 return None
157async def claim(refresh_token: str) -> bool:
158 """Try to become the one request that exchanges this token.
160 Returns True for the winner — and ALSO True whenever Redis cannot answer,
161 so a cache outage degrades to today's behaviour instead of blocking logins.
162 """
163 client = await _get_client()
164 if client is None:
165 return True
166 try:
167 acquired = await client.set(
168 _claim_key(refresh_token), "1", nx=True, ex=_CLAIM_TTL_SECONDS
169 )
170 return bool(acquired)
171 except Exception:
172 logger.warning(
173 "Single-flight refresh: could not place a claim; proceeding without it.",
174 exc_info=True,
175 )
176 return True
179async def publish(refresh_token: str, payload: dict[str, Any]) -> None:
180 """Hand the winner's result to any sibling still waiting on it."""
181 client = await _get_client()
182 if client is None:
183 return
184 try:
185 await client.set(
186 _result_key(refresh_token), json.dumps(payload), ex=_RESULT_TTL_SECONDS
187 )
188 except Exception:
189 logger.warning(
190 "Single-flight refresh: could not publish the result; a racing "
191 "sibling will fall through to Auth0.",
192 exc_info=True,
193 )
196async def await_result(
197 refresh_token: str, timeout: float = _WAIT_TIMEOUT_SECONDS
198) -> Optional[dict[str, Any]]:
199 """Wait for the winner's result. None means "go do it yourself"."""
200 client = await _get_client()
201 if client is None:
202 return None
204 loop = asyncio.get_event_loop()
205 deadline = loop.time() + timeout
206 while loop.time() < deadline:
207 try:
208 raw = await client.get(_result_key(refresh_token))
209 except Exception:
210 logger.warning(
211 "Single-flight refresh: could not read the result; falling through.",
212 exc_info=True,
213 )
214 return None
215 if raw:
216 try:
217 return json.loads(raw)
218 except (TypeError, ValueError):
219 logger.warning("Single-flight refresh: unreadable result payload.")
220 return None
221 await asyncio.sleep(_POLL_INTERVAL_SECONDS)
223 logger.info(
224 "Single-flight refresh: the in-flight rotation produced no result within "
225 "%ss; falling through to Auth0.",
226 timeout,
227 )
228 return None
231async def abandon(refresh_token: str) -> None:
232 """Drop a claim whose exchange failed, so a genuine retry is not blocked."""
233 client = await _get_client()
234 if client is None:
235 return
236 try:
237 await client.delete(_claim_key(refresh_token))
238 except Exception:
239 logger.warning(
240 "Single-flight refresh: could not release a failed claim; it expires "
241 "in %ss.",
242 _CLAIM_TTL_SECONDS,
243 exc_info=True,
244 )