Interactive Showcase: AI Reflex Rush
A real-time stress test of a fast multimodal LLM's instant binary decision-making capabilities, wrapped in a polished cyberpunk runner game. The same low-latency control pattern is what I use to evaluate Azure-deployed Copilot agents for production responsiveness.
Project details
The Concept: Instinctive AI & The OODA Loop
Typical Generative AI applications rely on 'Chain of Thought' reasoning, taking time to deliberate before answering. While powerful, this is too slow for real-time control systems. AI Reflex Rush is an experiment in 'Instinctive AI'.
In military strategy, the OODA Loop (Observe, Orient, Decide, Act) describes the decision cycle. Standard AI agents spend significant time in the 'Orient' phase, processing context and reasoning. By using a fast multimodal LLM with thinking disabled, we bypass the 'Orient' phase, forcing the model to go directly from Observe (Game State) to Decide (Jump/Wait), effectively simulating a biological reflex.
To achieve this, the model employs Spatial Reasoning. Instead of seeing a video, it processes a compressed vector representation of the game world, calculating the delta between the player's position and the nearest hurdle at a semantic level.
The Engineering Challenge: The Latency Gap
A standard web game runs at 60 frames per second (approx. 16ms per frame). An API call to a cloud-based LLM typically takes between 200ms and 600ms. If the game engine waited for the AI to decide, the game would freeze for half a second at every obstacle, making it unplayable.
To solve this, I implemented an Asynchronous Control Loop with Predictive Injections. The game engine runs independently of the AI. When an obstacle enters the 'Danger Zone', a snapshot is sent to the LLM. The model doesn't just react to the current frame; its system prompt instructs it to predict if a jump will be needed in 300ms. When the 'JUMP' signal arrives, it's executed regardless of the current frame, successfully bridging the cloud latency gap using purely generative prediction.
Real-World Applications: High-Velocity Semantic Control
This project is more than a game; it proves that LLMs can be used for High-Velocity Semantic Control. By stripping away complex reasoning, we can use the model's semantic understanding for real-time industrial and safety decisions.
Industrial Sorting & QA
Imagine a conveyor belt with mixed waste. A camera feeds images to the model. Instead of 'jumping', the model outputs 'PLASTIC' or 'METAL' instantly. The semantic understanding allows it to recognize a crushed soda can as metal, even if the shape is irregular—something traditional CV struggles with.
Live Content Moderation
For live-streaming chat, latency matters. This architecture can classify a comment as 'SAFE' or 'TOXIC' in milliseconds, blocking harmful content before it appears on screen without delaying the conversation.
Smart IoT Triggers
Security cameras often trigger false alarms for moving trees. A low-latency LLM can process the video frame and output a binary 'THREAT' / 'IGNORE' decision based on semantic context (e.g., 'A person in a mask' vs. 'A dog running'), reducing notification fatigue.
Prompt Engineering: Handling Verticality
Early versions of the AI simply jumped at every object. This caused a critical failure mode: jumping into high-flying drones. To fix this, the prompt had to be engineered to understand game physics and object classifications.
The system now explicitly categorizes threats into `GROUND` (spikes/blocks) and `AERIAL` (drones). The prompt logic enforces a rule: threats on the ground require a jump, but threats in the air require the player to stay grounded.
<span class="code-keyword">const</span> prompt = `<span class="code-class">You</span> are an expert <span class="code-class">AI</span> gamer playing a side-scrolling runner game.
<span class="code-class">RULES</span>:
<span class="code-number">1</span>. <span class="code-class">If</span> the obstacle is a <span class="code-string">'<span class="code-class">GROUND</span>'</span> object, you <span class="code-class">MUST</span> <span class="code-string">'<span class="code-class">JUMP</span>'</span>.
<span class="code-number">2</span>. <span class="code-class">If</span> the obstacle is a <span class="code-string">'<span class="code-class">FLYING</span>'</span> object, you <span class="code-class">MUST</span> <span class="code-string">'<span class="code-class">WAIT</span>'</span> (run under it).
<span class="code-class">Your</span> only possible actions are <span class="code-string">'<span class="code-class">JUMP</span>'</span> or <span class="code-string">'<span class="code-class">WAIT</span>'</span>.
<span class="code-class">Based</span> on the following state, respond with <span class="code-class">ONLY</span> <span class="code-string">'<span class="code-class">JUMP</span>'</span> or <span class="code-string">'<span class="code-class">WAIT</span>'</span>.
<span class="code-class">Game</span> <span class="code-class">State</span>: ${gameState}`;Optimization: Disabled Thinking Mode
The most critical API configuration disables the model's internal 'thinking' (the reasoning steps used for complex math or logic). By forcing the model to output the next token immediately, we shed valuable milliseconds off Time To First Token, making real-time gameplay possible. The same low-latency configuration applies to Azure OpenAI deployments where strict response budgets matter.
