> ## Documentation Index > Fetch the complete documentation index at: https://docs.omophub.com/llms.txt > Use this file to discover all available pages before exploring further. # Custom Concept Grouping for Cohorts > Build custom concept groupings on top of the fixed OMOP hierarchy using concept set expressions - include a grouper, expand its descendants, exclude what does not belong. ## 1. The Problem: "The Hierarchy Is Not My Cohort" You resolve clinical text to OMOP concepts and now you want to analyze them as a group. Say you want a cohort of patients with **abnormal findings in the lung**. Your pipeline surfaced two concepts: | Concept ID | Name | Concept class | | ---------- | ----------------------- | ---------------- | | `4142875` | Solitary nodule of lung | Clinical Finding | | `4116778` | Lesion of lung | Disorder | Both belong in the cohort. Neither is an ancestor of the other. And the OMOP hierarchy is fixed - you cannot add a "my abnormal lung findings" node to `concept_ancestor`. So it looks like a binary choice: accept the vocabulary's predefined groupings, or hand-enumerate every relevant concept ID forever. **It is not a binary choice.** There is a third option, and it is the one OHDSI tooling (ATLAS, Circe) is built on: a **concept set expression**. ## 2. The Core Concept: Store the Recipe, Not the List A concept set expression is a small, declarative recipe: ```json theme={null} { "include": [{ "concept_id": 4115260, "include_descendants": true }], "exclude": [{ "concept_id": 37311178, "include_descendants": true }] } ``` You store *that* - four or five lines - not the two thousand concept IDs it resolves to. At query time you **expand** it against the hierarchy to get the concrete list. This flips the mental model. The predefined hierarchy is not the constraint - it is the **expansion engine**. Your custom grouping is the expression you layer on top of it. Expressions are small enough to review in a pull request, version, and diff. A hand-maintained list of 2,000 concept IDs is none of those things, and it silently rots every time the vocabulary is released. The OMOPHub API is read-only: it expands expressions, it does not store them. Persistence, and the union/intersect/exclude set algebra, live in your application. That is a handful of Python `set` operations - see the recipe below. ## 3. Step One: Find the Grouper (Do Not Guess It) The instinct is to eyeball a parent concept. **Resist it.** For our lung example, the obvious guess is `257907` "Disorder of lung" - and it is wrong: | Candidate grouper | Descendants | Covers `4116778` (Lesion) | Covers `4142875` (Nodule) | | ------------------------- | ----------- | ------------------------- | ------------------------- | | `257907` Disorder of lung | 2,408 | ✅ | ❌ **missed** | | `4115260` Lung finding | 2,469 | ✅ | ✅ | "Disorder of lung" misses "Solitary nodule of lung" entirely, because that concept is a SNOMED **Clinical Finding**, not a **Disorder**. A cohort built on the intuitive grouper would silently drop every patient whose lung abnormality was coded as a finding. Compute the grouper instead. Fetch the ancestors of each seed concept and intersect them - the shared ancestors, nearest first, are your candidates: ```python theme={null} PAGE_SIZE = 200 # the endpoint clamps anything larger to 200 MAX_LEVELS = 20 # the endpoint's maximum; it defaults to only 10 def fetch_all(path, key): """Read every page of a paginated hierarchy response. Three separate caps can silently shrink a hierarchy result. All three have to be handled or your concept set is quietly incomplete: * `page_size` - one call returns a single page. Page until has_next is False. * `max_levels` - defaults to 10, so descendants deeper than 10 levels are dropped. Nothing in the response flags this: `truncated` reports the row cap only. Ask for the maximum (20). * `max_results` - the row cap. `truncated` does flag this one. """ page, out = 1, [] while True: body = client.get( path, max_levels=MAX_LEVELS, max_results=5000, include_invalid=False, page=page, page_size=PAGE_SIZE, ) out.extend(body["data"][key]) # note: payload is nested under "data" summary = body["data"]["hierarchy_summary"] if summary.get("truncated"): raise RuntimeError( f"{path} exceeds {summary['result_limit']} rows and was truncated. " "Narrow it with domain_ids / vocabulary_ids, or split it into " "several include seeds." ) if not body["meta"]["pagination"]["has_next"]: return out page += 1 def ancestors(concept_id): return fetch_all(f"/v1/concepts/{concept_id}/ancestors", "ancestors") def descendants(concept_id): return fetch_all(f"/v1/concepts/{concept_id}/descendants", "descendants") def find_groupers(seed_ids): """Ancestors shared by ALL seeds, tightest first. Each candidate is scored by its distance to the seed it is *furthest* from, then by its total distance. Ranking on a single seed's `min_levels_of_separation` would make the answer depend on which seed you happened to process last - the same candidate sits at different distances from different seeds. Seeds are de-duplicated first. `distances[candidate]` is keyed by seed, so a repeated seed collapses to one key while `len(seeds)` still counts it twice - the "shared by every seed" test could then never pass and this would return an empty list. Extraction pipelines produce duplicates routinely, whenever two phrases resolve to the same concept. """ seeds = sorted(set(seed_ids)) distances, meta = {}, {} for cid in seeds: for a in ancestors(cid): distances.setdefault(a["concept_id"], {})[cid] = a["min_levels_of_separation"] meta[a["concept_id"]] = a shared = [c for c, d in distances.items() if len(d) == len(seeds)] return sorted( ( {**meta[c], "worst_distance": max(distances[c].values())} for c in shared ), key=lambda a: ( a["worst_distance"], sum(distances[a["concept_id"]].values()), a["concept_id"], # deterministic tie-break: candidates do tie on both ), ) ``` For our two seeds this returns: ```text theme={null} 4115260 Lung finding worst_distance=2 4115259 Lower respiratory tract finding worst_distance=3 4185503 Finding of region of thorax worst_distance=3 4227253 Viscus structure finding worst_distance=3 4024567 Respiratory finding worst_distance=4 ``` `worst_distance` is how far the candidate sits from the seed it is furthest from. The candidate with the smallest one is the tightest grouping that still covers every seed - here, `4115260` "Lung finding". Anything further up (`441840` "Clinical finding") is too broad to be useful. This intersection is also a **quality check on your extraction pipeline**. If the nearest shared ancestor of your seeds is something as generic as "Clinical finding", the terms your agent proposed are not clinically coherent, and the grouping you are about to build will be junk. ## 4. Step Two: Expand, Then Subtract Hierarchy expansion alone is **over-inclusive**. The descendants of "Lung finding" include: * `37311178` Normal lung * `4064736` Lung function testing normal * `40481136` Lungs in normal arrangement * `4300172` Chest percussion normal These are the *opposite* of an abnormal finding. A naive `WHERE condition_concept_id IN (descendants_of_4115260)` produces a cohort that includes patients whose lungs were explicitly documented as normal. This is precisely why an expression has an `exclude` arm. The full recipe: ```python theme={null} expression = { "include": [ {"concept_id": 4115260, "include_descendants": True}, # Lung finding ], "exclude": [ {"concept_id": 37311178, "include_descendants": True}, # Normal lung {"concept_id": 4064736, "include_descendants": True}, # Lung function testing normal {"concept_id": 40481136, "include_descendants": True}, # Lungs in normal arrangement {"concept_id": 4300172, "include_descendants": True}, # Chest percussion normal ], "domain_id": "Condition", "standard_only": True, } ``` Expanding it yields **2,452 concepts**, containing both original seeds and none of the normal-finding branches. Reusing `descendants()` from the previous step: ```python theme={null} def keep(concept, expression): """The domain / standard filters, applied to any concept in the set.""" if expression.get("standard_only") and concept["standard_concept"] != "S": return False if expression.get("domain_id") and concept["domain_id"] != expression["domain_id"]: return False return True def expand(expression): def resolve(rules): out = set() for rule in rules: # The rule's own root is subject to the same filters as its # descendants. Skipping it here is how a classification ('C') or # wrong-domain grouper leaks into a cohort it can never legally # appear in. root = client.get(f"/v1/concepts/{rule['concept_id']}")["data"] if keep(root, expression): out.add(root["concept_id"]) if rule.get("include_descendants"): out.update( d["concept_id"] for d in descendants(rule["concept_id"]) if keep(d, expression) ) return out return resolve(expression["include"]) - resolve(expression.get("exclude", [])) ``` Two things this helper gets right that a naive version does not: * **It pages.** `/descendants` returns one page per call. A helper that reads only the first response caps every grouper at 200 concepts - your cohort would quietly contain a fraction of what it should, and your `exclude` branches would mostly not resolve either. * **It filters the rule roots, not just their descendants.** "Lung finding" is a Clinical Finding in the `Condition` domain, so it survives. But a classification grouper (`standard_concept = 'C'`, see below) would be added unconditionally by a naive `out.add(rule["concept_id"])` and land in a cohort where it is not legal. The final `set` difference is the entire "set algebra" you need. Union two expressions by unioning their expansions; intersect them the same way. ## 5. Two Filters You Must Not Skip **Filter by domain.** Descendants of a Condition-domain grouper do not all stay in Condition. "Lung finding" expands into 2,452 `Condition` concepts *and* 17 `Observation` concepts. If you are populating a condition cohort, `domain_id` must match the CDM table you are querying, or your `IN` clause will contain concept IDs that can never appear in `condition_occurrence`. **Filter to standard concepts.** Only `standard_concept = 'S'` concepts appear in the `*_concept_id` columns of an OMOP CDM. The column holds exactly three values: | Value | Meaning | Safe to use as a grouper? | Safe in `condition_concept_id`? | | ------ | ---------------------------- | ------------------------------- | ------------------------------- | | `'S'` | Standard | ✅ | ✅ | | `'C'` | Classification | ✅ (this is what they are *for*) | ❌ **never** | | `NULL` | Non-standard (a source code) | ❌ | ❌ | Note the third value is **`NULL`**, not the string `'N'`. OMOP has no `'N'`, so `standard_concept = 'N'` matches nothing and `standard_concept != 'S'` silently drops every non-standard concept (`NULL` fails any comparison). Test for it with `IS NULL`. The `keep()` helper above sidesteps this by checking `!= "S"` in Python, where `None != "S"` behaves as you would expect - but the same expression in SQL does not. Classification concepts - SNOMED's higher-level groupers, ICD-10 chapters, ATC drug classes - are legitimate and often *excellent* `include` seeds. But they are never written to a CDM event table. Expanding a `C` grouper and forgetting to filter its expansion to `S` is the single most common OMOP cohort bug. ## 6. Putting It Together ```text theme={null} agent proposes terms │ ▼ resolve to seed concept_ids │ ▼ intersect /ancestors ──────► candidate groupers (nearest first) │ ▼ author expression {include, exclude} ◄── you version this │ ▼ expand → filter domain + standard_concept='S' │ ▼ concept_id list → WHERE condition_concept_id IN (...) ``` Re-expand whenever you adopt a new vocabulary release. The expression stays the same; the concept list it produces gets better. ## 7. Scaling and Gotchas **Batch your lookups.** `POST /v1/concepts/hierarchy/batch` takes up to 100 queries in a single request and counts as **one** call against your rate limit (2 req/s on Free). Expanding 500 groupers is five requests, not five hundred. ```json theme={null} { "queries": [ { "query_id": "a", "concept_id": 4115260, "operation": "descendants", "params": { "max_results": 5000 } } ] } ``` **Set `max_results` explicitly on very large groupers.** The ceiling is 5,000 descendants per concept. If a grouper is broad enough to approach that, split it into several narrower `include` seeds rather than relying on one enormous root. **Set `max_levels=20` explicitly, always.** It defaults to `10`, and `truncated` does **not** flag depth truncation - only the row cap. A grouper whose tree runs deeper than 10 levels silently loses its deepest descendants while the response still reports `truncated: false`. This is the easiest way to build a quietly incomplete cohort. (A handful of SNOMED trees run deeper than 20, which is the API maximum; for those, seed from a lower node in the tree.) **Check the true descendant count before you expand.** `GET /v1/concepts/{concept_id}/level` returns `total_descendants` without fetching the set - a cheap way to catch a grouper that is far broader than you intended. **`include_invalid` defaults to `false`.** Deprecated concepts are excluded unless you ask for them. That is almost always what you want for a cohort; be deliberate if you override it. ## 8. Discovering What You Missed Hierarchy expansion only finds what the hierarchy already connects. To catch concepts your agent never proposed: * **`POST /v1/concepts/recommended`** implements the OHDSI **Phoebe** algorithm. Feed it your seed set and it suggests concepts that co-occur in real concept sets - a direct answer to "what am I forgetting?" * **`GET /v1/search/semantic`** with `standard_concept=S` and `domain_ids=Condition` surfaces concepts worded differently from your extracted terms. Feed anything promising back in as a new `include` seed and re-expand. The expression grows; the maintenance burden does not.