Visual Search

Visual Search finds AdventureQuest Worlds items by how they look. Describe an item in words, give it a picture, or point at items it already knows and ask for more like them. It also answers what goes with an outfit: given the pieces someone is wearing and an empty slot, it ranks that slot's items by style and colour. Every search comes back as a ranked list and a numbered collage, where cell 7 is result 7.

It is a small Flask service with a CLIP index behind it. In production a Discord bot's /search command and the AQ Closet site call it over HTTP.

What is not here. The production item pictures come from a private Flash renderer, and the pictures themselves are Artix Entertainment's art, so neither is in this repository. mock_render.py stands in for the renderer with generated pictures of invented items, so the whole pipeline runs from a fresh clone. See What is not in this repository.

---

What it does

• Text search. A query such as "red dragon cape" is ranked by four signals, fused by rank rather than by score: CLIP's match between the words and each item's picture, the item names, the look of the items whose names match, and the AQW Wiki's description of each item. A theme word also reaches the words item names use for it (angel finds seraphic, halo and winged things just below the items called Angel), from aqw_visual_search/concepts.json. A word that names a slot or weapon type narrows the pool, so "legion armor" ranks only armors.
• Image search. An uploaded picture, or a link to a Discord attachment, is embedded with CLIP and ranked by look.
• Items like these. like_ids ranks by the indexed items' own vectors, so it loads no model and answers even while the model is unloaded. Several ids average into one look, and the seeds never answer themselves.
• Goes with. POST /items/visual/match takes the worn items and the slot to fill. Each render's CLIP vector has its slot's mean taken out, which leaves style rather than shape and compares across slots, and a colour signature of each render (hue, saturation and value over its visible pixels) is compared beside it. The two are z-scored and fused, style 0.6 and colour 0.4. No model is loaded.
• Filters. Slot and weapon-type tags (helm, armor, cape, weapon, pet, ground, then sword, dagger, staff and the other weapon types), rarity and availability words, colour-custom, and which body an armor is drawn on.
• The collage. Results come in blocks of 25, each with a PNG grid numbered to match the list, on a dark, white or any hex background. Paging reuses the ranked pool instead of searching again.
• Two bodies. Armors are rendered on the male and the female body and both renders are indexed. gender picks one, or lets both compete and says which one ranked each result.

How it fits together

1. A caller sends a query to POST /items/visual.
2. Words and pictures are embedded with CLIP. The model loads on the first search that needs it and is unloaded after 15 idle minutes. like_ids and goes-with use vectors already in the index.
3. The index ranks every item's render, narrowed by tags, rarity and body.
4. The ranked pool is cached under a query id (qid), and each block of 25 gets its collage when it is asked for.
5. Everything the service reads is in one data folder, built by the sync pipeline below. The service notices a new index file and reloads it about two minutes after the build finishes.

Built with Python 3.12, Flask and gunicorn, sentence-transformers' CLIP on CPU torch, NumPy and Pillow.

Routes

Route	What it does	Token
POST /items/visual	The search. JSON, or a multipart form with an image file.	if set
POST /items/visual/match	Goes-with: like_ids (the worn items), slot, and optional gender and count (1 to 200, default 40).	if set
GET /items/visual/collage/<qid>/<page>.png	The numbered grid for one block of a search. ?gender=M or F draws every cell on one body. The search hands this back as collage_url.	if set, header or ?t=
GET /items/visual/thumb/<id>.png	One item's render. ?gender=F gives the female one where it exists.	if set, header or ?t=
GET /items/search?q=&slot=&limit=	A name search over the item sheet, with no index or model, for a caller to fall back on while the index is not ready.	if set
GET /items/visual/rarities	The rarity values and counts the filters know, and the facts feed's state.	never
GET /health	Whether the index is loaded, its size and model, whether the CLIP model is resident and how long it took to load, installed versions, memory, and the state of the sync and the facts feed.	never
GET /items/visual/status?id= or ?name=	One item's pipeline record: whether it rendered, when, and if not, why.	required, header or ?t=
GET /items/visual/failures	Every item that did not render, with its error. ?slot= narrows it; ?suspect_bytes=3000 also lists renders small enough to be blank.	required, header or ?t=
GET /admin/sync?token=	Starts a sync run in the background, one at a time. Progress goes to the log.	required

"If set" routes check the X-Render-Token header against AQW_RENDER_TOKEN when that variable is set, and are open when it is not, so a local run needs no token. "Required" routes answer 403 until it is set.

The search request

At least one of description, image, image_url and like_ids is needed.

Field	Meaning
description	Words to search for.
image	A picture to search by (multipart only).
image_url	An https link to a picture on a host in AQW_IMAGE_URL_HOSTS (Discord's CDN by default). Redirects are refused.
like_ids	Item ids to find lookalikes of: one id, a comma list or a JSON list, up to 16. like_id works too.
tags	Slot or weapon-type tags, as a list or a comma string.
rarity	Filter words, combined with commas. Any rarity the wiki uses (epic, legendary) always works, and so do no-rares, no-seasonals and no-rares-seasonals (without the facts feed, only on their own). The availability words ingame, seasonal, rare, unobtainable and unknown, and ioda and kbioda, need the facts feed (below); without it each word is matched against the wiki's rarity names, and one that matches none is ignored. An item that matches any positive word passes, and -word excludes.
cc	true for items that take the player's colours, false for the rest.
gender	any (the default, both bodies compete), M or F.
search_type	auto (the default), text or image.
background	The collage background: dark (the default), white or a #hex.
qid, page	A previous answer's qid and a page number, to page through the same ranking.

The answer, shortened:

{
  "ready": true, "query_kind": "text", "term": "red cape",
  "total": 150, "page": 0, "pages": 6, "per": 25, "qid": "3f0c9a1e5b7d2c48",
  "collage_url": "/items/visual/collage/3f0c9a1e5b7d2c48/0.png?bg=dark",
  "filter": {}, "female_any": true,
  "results": [
    {"n": 1, "id": 900007, "name": "Cinderquill Mantle", "slot": "cape",
     "tags": ["cape"], "file": "items/capes/MockCinderquillMantle.swf",
     "wiki_url": "http://aqwwiki.wikidot.com/cinderquill-mantle",
     "score": 0.0476, "gender": "M", "female": false,
     "rarity": "", "state": "", "seasonal": false, "ioda": "", "cc": false}
  ]
}

ready: false with no error means the index is not loaded yet, and a caller can fall back to /items/search. rarity, state, seasonal and ioda are empty unless the facts feed (below) is configured. filter says which rarity words were applied and where the answer came from, so a filter that did nothing says so.

The data pipeline

python3 -m aqw_visual_search.sync_all --out <data folder> brings the data folder up to date with the item sheet. Every stage keeps a manifest and skips what it has already done, so a run after an AQW release only does the new items. The stages, in order:

1. Item sheet. itemdb_client.py reads a community-maintained sheet of AQW items (AQW_ITEMDB_URL), normalises each row to id, name, SWF path, slot, AC and member flags, and caches it (AQW_ITEMDB_CACHE).
2. Renders. collect_renders.py asks the render service at AQW_RENDER_URL for a picture of every item, one at a time by default, and saves thumbs/<id>.png. Armors also get thumbs/<id>F.png on the female body. Failures are retried up to three times and then parked with their error. While AQW_RENDER_URL is unset the stage is skipped with one log line.
3. Wiki notes. collect_wiki_meta.py reads each item's page on the AQW Wiki for its rarity, whether it mentions Color Custom, and its description, at about one request a second. A full pass takes several hours to a day. Touch STOP_META in the data folder to stop it; the next run continues.
4. Index. visual_index.py embeds every render with CLIP into visual_index.npz. The CLIP weights are downloaded at run time from Hugging Face on the first build (the default, clip-ViT-L-14, is about 1.7 GB) and read from the local cache after that. The service needs the same weights to embed text and picture queries, and fetches them the same way if its cache lacks them. Changing AQW_CLIP_MODEL re-embeds everything on the next build; searches keep using the model the loaded index was built with until then.
5. Colour. match.py writes a colour signature of every render to visual_colour.npz, for goes-with. Pillow only, a minute's work. Without it, goes-with answers on style alone and says so in channels.

--skip-renders, --skip-meta and --skip-index run a subset; --skip-index skips the colour stage too, which runs on its own as python3 -m aqw_visual_search.match --cache <data folder>. --only armor renders one slot, --refetch re-renders done items, --refetch-all retries parked ones, --suspect-bytes 3000 re-renders suspiciously small ones, and --no-female or --gender F change the female pass. Each stage also runs on its own as python3 -m aqw_visual_search.<module>; --help lists its options.

A sync starts three ways: from the command line, from /admin/sync (for a host with no shell), or on a schedule when AQW_AUTO_SYNC_DAYS is set.

collect_thumbs.py is an older alternative to the render stage that takes one picture per item from its wiki page instead. sync_all does not run it.

The facts feed (optional). With AQW_CLOSET_URL set, the rarity, availability, IoDA and colour-custom answers come from an AQ Closet site's GET /api/items/facts, refreshed every six hours and cached in the data folder. Without it, rarity words and colour-custom answer from the wiki notes and the file names, and the availability and IoDA words are only matched against the wiki's rarity names. A closet_facts.json already in the data folder is still read, but never refreshed. The feed is JSON keyed by the item sheet's ids:

{"v": 1, "capturedAt": <ms>, "count": N,
 "facts": {"<id>": {"r": "<the wiki's rarity word>",
                    "s": "in-game | seasonal | rare | unobtainable | unknown",
                    "z": 1, "i": "normal", "c": 1}}}

s is the availability band, z is 1 for an item that comes back with an event, i names the IoDA token that reaches it (normal, kb or empty), and c is 1 for an item that takes the player's colours.

The data folder after a sync:

item_db.json        the item sheet, cached (wherever AQW_ITEMDB_CACHE points)
manifest.json       one row per item: done, failed or skipped, and why
thumbs/<id>.png     one render per item; armors also have <id>F.png
wiki_meta.json      rarity, colour-custom and description per item
visual_index.npz    the CLIP vectors, with the model that made them
visual_colour.npz   the colour signatures
closet_facts.json   the last copy of the facts feed
auto_sync.json      when the scheduled sync last ran, and how it went

The render service

The renderer is a separate service, and anything that answers this request can be one:

GET <AQW_RENDER_URL>/render-item?file=<SWF path>&slot=<slot>&type=<weapon type>&gender=F&transparent=1
X-Render-Token: <AQW_RENDER_TOKEN>

slot is one of helm, armor, cape, weapon, pet and ground. type comes only with weapons, gender=F only on the female pass, and the token, when there is one, comes as the header and as ?t=. A success is a 200 with a PNG. An error is JSON {"detail": "..."} with an error status, and its detail is kept in the manifest. A PNG with nothing drawn counts as a failure.

What is not in this repository

• The renderer. Production renders come from a private Flash renderer that loads each item's SWF from the game and draws it on a body. It is not published. mock_render.py takes its place: it answers the same request with the same errors and draws a placeholder for each item whose shape follows the slot and whose colours follow the file name, so every item gets its own picture and its own vector.
• The art. The renders, the wiki images, and everything made from them (the thumbnails, the CLIP index, the colour signatures, the collages) are Artix Entertainment's art. They stay off the repository. data/ is git-ignored, so a clone starts without one and builds its own.
• The item list. It is fetched from the community sheet at run time and cached in the data folder.
• The CLIP weights. They are downloaded at run time from their publishers (see NOTICE).

The tests use invented items (tests/data/mock_items.json) and a list of AQW Wiki page titles (tests/data/wikidot_slugs.tsv) to check the slug rule.

Running it locally

You need Python 3.12. The service needs flask and torch; the tests and the stand-in renderer need only NumPy, Pillow and requests.

python3 -m venv .venv && . .venv/bin/activate
# CPU torch first: the default PyPI wheel is the CUDA build, about 4 GB more
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -r requirements.txt

1. Start the stand-in. It serves renders at /render-item and 21 invented items at /sheet.

python3 mock_render.py --port 8080

2. Build a data folder from it, in a second shell. AQW_ITEMDB_CACHE points into the same folder so the mock items do not replace a real item cache in ./data. --skip-meta because the invented items have no wiki pages.

export AQW_RENDER_URL=http://127.0.0.1:8080
export AQW_ITEMDB_URL=http://127.0.0.1:8080/sheet
export AQW_ITEMDB_CACHE=./mock_data/item_db.json
python3 -m aqw_visual_search.sync_all --out ./mock_data --skip-meta

It renders 24 pictures (21 items, and three armors again on the female body), downloads the CLIP weights on the first run, builds the index, and writes the colour signatures. For a smaller download, export AQW_CLIP_MODEL=clip-ViT-B-32 first (about 600 MB). Without torch the render and colour stages still finish, the index stage fails with ModuleNotFoundError, and sync_all exits 1 after saying which stage failed. Then this builds an index of placeholder vectors, enough for like_ids, goes-with and the collages:

python3 -m aqw_visual_search.visual_index --cache ./mock_data \
  --index ./mock_data/visual_index.npz --stub

Text and picture searches need torch and a real build, which re-embeds everything.

3. Serve it, in the same shell:

export AQW_THUMB_CACHE=./mock_data
export AQW_VISUAL_INDEX=./mock_data/visual_index.npz
export AQW_WIKI_META=./mock_data/wiki_meta.json
python3 app.py

4. Ask it. The service listens on port 8000.

curl -s localhost:8000/items/visual -H 'Content-Type: application/json' \
  -d '{"description": "red cape"}'
curl -s localhost:8000/items/visual -F image=@mock_data/thumbs/900007.png
curl -s localhost:8000/items/visual/match -H 'Content-Type: application/json' \
  -d '{"like_ids": [900004], "slot": "cape"}'

Open the collage_url from an answer on http://localhost:8000 to see the grid. The placeholders are flat shapes in flat colours, so the rankings over them say little about real art; they show that every part runs.

To try the token checks, start the stand-in with MOCK_RENDER_TOKEN=x, set AQW_RENDER_TOKEN=x for the pipeline and the service, send -H 'X-Render-Token: x' with each search, and add &t=x to a collage_url opened in a browser.

Tests

Each test_*.py is a plain script that exits non-zero on a failure. None of them needs flask, torch, a model download or internet access; test_mock_render.py starts the stand-in on localhost.

for f in test_*.py; do python3 "$f" || echo "failed: $f"; done

.forgejo/workflows/checks.yml runs every one on each push and pull request.

Deploying

The Dockerfile builds the image the service runs from in production, on Railway; any host that gives the container a persistent volume will do. It installs CPU torch, runs gunicorn with one worker and four threads on port 8000, and expects a volume at /app/data, where it points the data folder, the item cache and HF_HOME, so the CLIP weights download once rather than on every deploy. One worker matters: the model and index are per process, and a second worker would run a second sync scheduler. Plan for about 2 GB of RAM: the default model peaks at about 1.7 GB while it is loaded. Gunicorn binds [::]:8000; on a host without IPv6, bind it to 0.0.0.0:8000 instead (python3 app.py falls back to IPv4 on its own).

On a fresh volume, set AQW_RENDER_TOKEN and AQW_RENDER_URL, then open /admin/sync?token=<token> once to build the index, and set AQW_AUTO_SYNC_DAYS to keep it current. The first run renders every item and reads every item's wiki page before it builds the index, so the first index lands several hours to a day later. On a host with a shell, python3 -m aqw_visual_search.sync_all --out /app/data --skip-meta builds one sooner, and a later sync fills in the wiki notes. /health shows when the new index lands.

Configuration

Every variable the code reads. Unset, each takes the default shown. .env.example lists them all, blank.

The service

Variable	Default	What it does
PORT	8000	The port for python app.py. The Dockerfile's gunicorn binds 8000 itself.
AQW_RENDER_TOKEN	unset	The shared secret. Set, the search routes want it in X-Render-Token (or ?t= on the image routes), the admin and debug routes open with it, and the render stage sends it to the render service. Unset, search is open and the admin and debug routes answer 403.
AQW_PUBLIC_BASE	unset	The service's public address, put in front of collage_url so it is absolute. Unset, collage_url is a path.
AQW_MAX_BODY_BYTES	16777216	The largest request body, in bytes (16 MB).
AQW_MAX_IMAGE_PIXELS	178956970	The largest query image, in pixels, read from its header before it is decoded. Larger is a 400 that names the size. It can lower Pillow's own limit, not raise it.
AQW_IMAGE_URL_HOSTS	cdn.discordapp.com,media.discordapp.net	The hosts an image_url may name, and their subdomains.
AQW_IMAGE_URL_MAX_BYTES	8388608	The most an image_url fetch reads, in bytes (8 MB).
AQW_COLLAGE_PAGE_CACHE	2	Collage pages kept per query, for the last 16 queries.

The data folder

Variable	Default	What it does
AQW_THUMB_CACHE	/app/data/thumb_cache for search, /app/data for the sync	The data folder: where search reads renders, the manifest and the facts cache, and where /admin/sync and the schedule write. The two defaults differ, so set it; the Dockerfile sets /app/data.
AQW_VISUAL_INDEX	/app/data/visual_index.npz for search	The index file. Unset, sync_all writes <out>/visual_index.npz and visual_index run on its own writes ./thumb_cache/visual_index.npz.
AQW_COLOUR_INDEX	<data folder>/visual_colour.npz	The colour signatures for goes-with.
AQW_WIKI_META	/app/data/wiki_meta.json	The wiki notes file search reads for descriptions and the rarity fallback.
AQW_ITEMDB_CACHE	./data/item_db.json	Where the item sheet is cached. The Dockerfile sets /app/data/item_db.json.

The model

Variable	Default	What it does
AQW_CLIP_MODEL	clip-ViT-L-14	The CLIP model the index is built with, by its sentence-transformers name, or openclip:<arch>/<pretrained> with open_clip_torch installed. Search follows the model the loaded index records; it falls back to this variable, then to clip-ViT-B-32, only for an index that records none.
AQW_VISUAL_PREWARM_MODEL	0	1 loads the model at boot instead of on the first search.
AQW_MODEL_IDLE_S	900	Seconds without a search before the model is unloaded, checked once a minute. The next search loads it again from the local cache.
AQW_TORCH_THREADS	2	Torch's CPU threads.
AQW_MEM_LOG_S	3600	Seconds between [mem] log lines (process RSS and the container's memory split). 0 turns off the periodic lines; one is still printed at boot. A value that is not a number keeps the default and says so in the log.

The item sheet

Variable	Default	What it does
AQW_ITEMDB_URL	the community sheet	The item list, as a JSON array of rows. The default is a community-maintained Google Sheet read through opensheet (see itemdb_client.py).
AQW_ITEMDB_VERBOSE	unset	1 logs every contested item id when the sheet loads, instead of a sample.
AQW_USE_HOST_ITEMDB	0	1 uses an itemdb.py on PYTHONPATH with a get_db() instead of the built-in client, for embedding this in a host application.

The render service

Variable	Default	What it does
AQW_RENDER_URL	unset	The render service's base address. Unset, the render stage is skipped.
AQW_RENDER_TIMEOUT	120	Seconds to wait for one render.
AQW_RENDER_TRANSPARENT	1	Asks for renders on a transparent background. 0, false or no asks for the service's own backdrop.
AQW_RENDER_USER_AGENT	aqw-visual-search/1.0 (swf render cache)	The User-Agent sent to the render service.

The AQW Wiki

Variable	Default	What it does
AQW_WIKI_BASE	http://aqwwiki.wikidot.com	The wiki the collectors read. Nothing fetches it until a sync runs.
AQW_WIKI_ITEM_BASE	http://aqwwiki.wikidot.com	The wiki address in each result's wiki_url.
AQW_WIKI_META_USER_AGENT	aqw-visual-search/1.0 (wiki metadata: rarity, cc)	The User-Agent for the wiki notes crawl. A public crawl should say who runs it, so put a contact of your own here.
AQW_THUMBS_USER_AGENT	aqw-visual-search/1.0 (thumbnail cache builder)	The User-Agent for collect_thumbs.py.

The facts feed and the schedule

Variable	Default	What it does
AQW_CLOSET_URL	unset	An AQ Closet site whose /api/items/facts the rarity filters answer from. Unset, rarity words answer from the wiki notes, and a closet_facts.json already in the data folder is used as it is.
AQW_CLOSET_FACTS_REFRESH_S	21600	Seconds before the cached facts are fetched again (checked every ten minutes).
AQW_AUTO_SYNC_DAYS	unset (off)	Days between scheduled syncs. Off unless it is a positive number; the startup log says which, and why. The first run comes one period after the first boot, and a failed run is retried a day later, or one period later when the period is shorter.

The stand-in renderer

Variable	Default	What it does
MOCK_RENDER_TOKEN	unset	The token mock_render.py wants on /render-item. Unset, it uses AQW_RENDER_TOKEN; with neither, it wants none.

Where the model cache is

Hugging Face's libraries read these to decide where the CLIP weights are cached. The service reads them only to say, in the log and on /health (model_cache), where that is and whether it survives a redeploy.

Variable	Default	What it does
SENTENCE_TRANSFORMERS_HOME	unset	sentence-transformers' cache folder, if set.
HF_HUB_CACHE	unset	The Hugging Face hub cache.
HUGGINGFACE_HUB_CACHE	unset	The older name for HF_HUB_CACHE.
HF_HOME	unset	Hugging Face's home; the cache is its hub/. The Dockerfile sets /app/data/hf.
XDG_CACHE_HOME	unset	Used when none of the above is set; the cache is huggingface/hub under it, else under ~/.cache.
RAILWAY_VOLUME_MOUNT_PATH	set by Railway	The volume's mount point, to check the cache is on it.
RAILWAY_PROJECT_ID, RAILWAY_SERVICE_ID, RAILWAY_ENVIRONMENT_NAME, RAILWAY_ENVIRONMENT	set by Railway	Any of them means the service runs on Railway, where no mount point means no volume.

Licence

The code is MIT; see LICENSE. NOTICE covers what the licence does not: the AQW Wiki titles in the tests, the CLIP weights, and AdventureQuest Worlds itself. Visual Search is a fan project, not affiliated with or endorsed by Artix Entertainment.