> ## Documentation Index > Fetch the complete documentation index at: https://docs.omophub.com/llms.txt > Use this file to discover all available pages before exploring further. # Custom Concept Grouping for Cohorts > Build custom concept groupings on top of the fixed OMOP hierarchy using concept set expressions - include a grouper, expand its descendants, exclude what does not belong. ## 1. The Problem: "The Hierarchy Is Not My Cohort" You resolve clinical text to OMOP concepts and now you want to analyze them as a group. Say you want a cohort of patients with **abnormal findings in the lung**. Your pipeline surfaced two concepts: | Concept ID | Name | Concept class | | ---------- | ----------------------- | ---------------- | | `4142875` | Solitary nodule of lung | Clinical Finding | | `4116778` | Lesion of lung | Disorder | Both belong in the cohort. Neither is an ancestor of the other. And the OMOP hierarchy is fixed - you cannot add a "my abnormal lung findings" node to `concept_ancestor`. So it looks like a binary choice: accept the vocabulary's predefined groupings, or hand-enumerate every relevant concept ID forever. **It is not a binary choice.** There is a third option, and it is the one OHDSI tooling (ATLAS, Circe) is built on: a **concept set expression**. ## 2. The Core Concept: Store the Recipe, Not the List A concept set expression is a small, declarative recipe: ```json theme={null} { "include": [{ "concept_id": 4115260, "include_descendants": true }], "exclude": [{ "concept_id": 37311178, "include_descendants": true }] } ``` You store *that* - four or five lines - not the two thousand concept IDs it resolves to. At query time you **expand** it against the hierarchy to get the concrete list. This flips the mental model. The predefined hierarchy is not the constraint - it is the **expansion engine**. Your custom grouping is the expression you layer on top of it. Expressions are small enough to review in a pull request, version, and diff. A hand-maintained list of 2,000 concept IDs is none of those things, and it silently rots every time the vocabulary is released. The OMOPHub API is read-only: it expands expressions, it does not store them. Persistence, and the union/intersect/exclude set algebra, live in your application. That is a handful of Python `set` operations - see the recipe below. ## 3. Step One: Find the Grouper (Do Not Guess It) The instinct is to eyeball a parent concept. **Resist it.** For our lung example, the obvious guess is `257907` "Disorder of lung" - and it is wrong: | Candidate grouper | Descendants | Covers `4116778` (Lesion) | Covers `4142875` (Nodule) | | ------------------------- | ----------- | ------------------------- | ------------------------- | | `257907` Disorder of lung | 2,408 | ✅ | ❌ **missed** | | `4115260` Lung finding | 2,469 | ✅ | ✅ | "Disorder of lung" misses "Solitary nodule of lung" entirely, because that concept is a SNOMED **Clinical Finding**, not a **Disorder**. A cohort built on the intuitive grouper would silently drop every patient whose lung abnormality was coded as a finding. Compute the grouper instead. Fetch the ancestors of each seed concept and intersect them - the shared ancestors are your candidates, ranked by how many seeds they cover and how tightly: ```python theme={null} PAGE_SIZE = 200 # the endpoint clamps anything larger to 200 MAX_LEVELS = 20 # the endpoint's maximum; it defaults to only 10 def fetch_all(path, key): """Read every page of a paginated hierarchy response. Three separate caps can silently shrink a hierarchy result. All three have to be handled or your concept set is quietly incomplete: * `page_size` - one call returns a single page. Page until has_next is False. * `max_levels` - defaults to 10, so descendants deeper than 10 levels are dropped. Nothing in the response flags this: `truncated` reports the row cap only. Ask for the maximum (20). * `max_results` - the row cap. `truncated` does flag this one. """ page, out = 1, [] while True: body = client.get( path, max_levels=MAX_LEVELS, max_results=5000, include_invalid=False, page=page, page_size=PAGE_SIZE, ) out.extend(body["data"][key]) # note: payload is nested under "data" summary = body["data"]["hierarchy_summary"] if summary.get("truncated"): raise RuntimeError( f"{path} exceeds {summary['result_limit']} rows and was truncated. " "Narrow it with domain_ids / vocabulary_ids, or split it into " "several include seeds." ) if not body["meta"]["pagination"]["has_next"]: return out page += 1 def ancestors(concept_id): return fetch_all(f"/v1/concepts/{concept_id}/ancestors", "ancestors") def descendants(concept_id): return fetch_all(f"/v1/concepts/{concept_id}/descendants", "descendants") def find_groupers(seed_ids, min_coverage=None): """Shared ancestors of the seeds, best candidate first. Candidates are ranked by how many seeds they cover, then by their distance to the seed they are *furthest* from, then by total distance. Ranking on a single seed's `min_levels_of_separation` would make the answer depend on which seed you happened to process last - the same candidate sits at different distances from different seeds. `min_coverage` defaults to "every seed", but partial coverage is deliberately reported rather than discarded. One outlier seed otherwise drags the only surviving candidate all the way up to "Clinical finding"; a 4-of-5 candidate with 2,400 descendants is a far better grouper than a 5-of-5 candidate with 194,000, and you handle the fifth seed as its own include rule. Seeds are de-duplicated first. `distances[candidate]` is keyed by seed, so a repeated seed collapses to one key while `len(seeds)` still counts it twice - the coverage test could then never pass and this would return an empty list. Extraction pipelines produce duplicates routinely, whenever two phrases resolve to the same concept. """ seeds = sorted(set(seed_ids)) floor = min_coverage or len(seeds) distances, meta = {}, {} for cid in seeds: for a in ancestors(cid): distances.setdefault(a["concept_id"], {})[cid] = a["min_levels_of_separation"] meta[a["concept_id"]] = a return sorted( ( { **meta[c], "seeds_covered": len(d), "worst_distance": max(d.values()), "total_distance": sum(d.values()), } for c, d in distances.items() if len(d) >= floor ), key=lambda a: ( -a["seeds_covered"], a["worst_distance"], a["total_distance"], a["concept_id"], # deterministic tie-break: candidates do tie on both ), ) def with_cost(candidates, top_n=8): """Annotate the top candidates with what they would drag in. `/level` returns the count without fetching the set, so this is one cheap call per candidate. Note `total_descendants` comes back as a **string** - compare it as a number or the size cliff below will not sort the way you expect. """ for c in candidates[:top_n]: level = client.get(f"/v1/concepts/{c['concept_id']}/level")["data"] c["total_descendants"] = int(level["total_descendants"]) return candidates[:top_n] ``` For the two seeds above, plus three more that a real extraction run would surface - `4249557` Radiologic infiltrate of lung, `4200027` Cavitation of lung, `37206719` Nodule of lung - this returns: | Candidate | Seeds covered | Worst distance | `total_descendants` | | --------------------------------------- | ------------- | -------------- | ------------------- | | **4115260 Lung finding** | **5/5** | 3 | **2,469** | | 4115259 Lower respiratory tract finding | 5/5 | 4 | 3,409 | | 4024567 Respiratory finding | 5/5 | 5 | 13,995 | | 4185503 Finding of region of thorax | 5/5 | 4 | 14,469 | | 441840 Clinical finding | 5/5 | 5 | 194,735 | | 257907 Disorder of lung | 4/5 ❌ | 2 | 2,408 | Read two things off it. **The intuitive grouper stays wrong as the seed set grows.** `257907` "Disorder of lung" still misses a seed at five seeds, just as it did at two. More seeds do not rescue a grouper that sits on the wrong branch. **There is a size cliff, and it is your stopping rule.** One step up from "Lung finding" costs +940 concepts for zero extra coverage; the step after that is a 5.9x blow-up. Climbing all the way to "Clinical finding" costs 79x for nothing at all. So: **climb while coverage improves, and stop the moment the descendant count jumps disproportionately.** This intersection is also a **quality check on your extraction pipeline**. If the nearest shared ancestor of your seeds is something as generic as "Clinical finding", the terms your agent proposed are not clinically coherent, and the grouping you are about to build will be junk. ## 4. Step Two: Expand, Then Subtract Hierarchy expansion alone is **over-inclusive**. The descendants of "Lung finding" include: | Concept | Domain | | -------------------------------------- | ------------- | | `37311178` Normal lung | Observation | | `40481136` Lungs in normal arrangement | Observation | | `4300172` Chest percussion normal | Observation | | `4064736` Lung function testing normal | **Condition** | These are the *opposite* of an abnormal finding. A naive `WHERE condition_concept_id IN (descendants_of_4115260)` produces a cohort that includes patients whose lungs were explicitly documented as normal. Note the domain column, because it changes how much work each filter is doing. The domain filter in the next section removes the first three on its own. `4064736` "Lung function testing normal" is a **Condition**, so it survives every filter and lands in the cohort - which is exactly why the `exclude` arm is not optional. A normal finding really can be Condition-domain, and no amount of domain or standard-concept filtering will catch it. This is precisely why an expression has an `exclude` arm. The full recipe: ```python theme={null} expression = { "include": [ {"concept_id": 4115260, "include_descendants": True}, # Lung finding ], "exclude": [ {"concept_id": 37311178, "include_descendants": True}, # Normal lung {"concept_id": 4064736, "include_descendants": True}, # Lung function testing normal {"concept_id": 40481136, "include_descendants": True}, # Lungs in normal arrangement {"concept_id": 4300172, "include_descendants": True}, # Chest percussion normal ], "domain_id": "Condition", "standard_only": True, } ``` Expanding it yields **2,452 concepts**, containing both original seeds and none of the normal-finding branches. Reusing `descendants()` from the previous step: ```python theme={null} def keep(concept, expression): """The domain / standard filters, applied to any concept in the set.""" if expression.get("standard_only") and concept["standard_concept"] != "S": return False if expression.get("domain_id") and concept["domain_id"] != expression["domain_id"]: return False return True def expand(expression): def resolve(rules): out = set() for rule in rules: # The rule's own root is subject to the same filters as its # descendants. Skipping it here is how a classification ('C') or # wrong-domain grouper leaks into a cohort it can never legally # appear in. root = client.get(f"/v1/concepts/{rule['concept_id']}")["data"] if keep(root, expression): out.add(root["concept_id"]) if rule.get("include_descendants"): out.update( d["concept_id"] for d in descendants(rule["concept_id"]) if keep(d, expression) ) return out return resolve(expression["include"]) - resolve(expression.get("exclude", [])) ``` Two things this helper gets right that a naive version does not: * **It pages.** `/descendants` returns one page per call. A helper that reads only the first response caps every grouper at 200 concepts - your cohort would quietly contain a fraction of what it should, and your `exclude` branches would mostly not resolve either. * **It filters the rule roots, not just their descendants.** "Lung finding" is a Clinical Finding in the `Condition` domain, so it survives. But a classification grouper (`standard_concept = 'C'`, see below) would be added unconditionally by a naive `out.add(rule["concept_id"])` and land in a cohort where it is not legal. The final `set` difference is the entire "set algebra" you need. Union two expressions by unioning their expansions; intersect them the same way. ## 5. Two Filters You Must Not Skip **Filter by domain.** Descendants of a Condition-domain grouper do not all stay in Condition. "Lung finding" expands into 2,453 `Condition` concepts *and* 17 `Observation` concepts. If you are populating a condition cohort, `domain_id` must match the CDM table you are querying, or your `IN` clause will contain concept IDs that can never appear in `condition_occurrence`. **Filter to standard concepts.** Only `standard_concept = 'S'` concepts appear in the `*_concept_id` columns of an OMOP CDM. The column holds exactly three values: | Value | Meaning | Safe to use as a grouper? | Safe in `condition_concept_id`? | | ------ | ---------------------------- | ------------------------------- | ------------------------------- | | `'S'` | Standard | ✅ | ✅ | | `'C'` | Classification | ✅ (this is what they are *for*) | ❌ **never** | | `NULL` | Non-standard (a source code) | ❌ | ❌ | Note the third value is **`NULL`**, not the string `'N'`. OMOP has no `'N'`, so `standard_concept = 'N'` matches nothing and `standard_concept != 'S'` silently drops every non-standard concept (`NULL` fails any comparison). Test for it with `IS NULL`. The `keep()` helper above sidesteps this by checking `!= "S"` in Python, where `None != "S"` behaves as you would expect - but the same expression in SQL does not. Classification concepts - SNOMED's higher-level groupers, ICD-10 chapters, ATC drug classes - are legitimate and often *excellent* `include` seeds. But they are never written to a CDM event table. Expanding a `C` grouper and forgetting to filter its expansion to `S` is the single most common OMOP cohort bug. ## 6. When Your Pipeline Lands in the Wrong Domain Everything above assumes your seeds are in the domain you are analyzing. A text-extraction pipeline routinely breaks that assumption: it resolves a phrase to a concept with exactly the right *name* sitting in the wrong *domain*. The natural next question is whether some equivalence mapping can convert it back. **It cannot.** OMOP has no relationship that moves a concept to its equivalent in another domain, because there is no such thing - domain is a property of the concept, and two same-named concepts in different domains are genuinely different things sharing a string. Measured against release 2026.1: * **8,467** pairs of standard concepts share a name across different domains. **None** are linked by an equivalence relationship. The 765 that are linked at all are overwhelmingly `Has asso morph` - a semantic attribute, not equivalence. * Every `eq` relationship in the vocabulary (`SNOMED - CPT4 eq`, `LOINC - SNOMED eq`, `RxNorm - ATC name`, ...) crosses **vocabularies**, never domains. * Of the **3,518,557** `Maps to` rows that originate from a standard concept, **100% point at the concept itself.** Once you hold a standard concept, `Maps to` is a no-op. It relocates you only when the source is a non-standard code. ### Why this is worse than a mislabeled domain Searching for "chronic obstructive pulmonary disease" without a domain filter returns two concepts with identical names: ```text theme={null} 255573 Condition Chronic obstructive pulmonary disease Disorder 36308250 Meas Value Chronic obstructive pulmonary disease Answer ← rank 2 ``` `36308250` is a LOINC answer-list value - what someone ticked on a questionnaire, not a diagnosis. It has no relationship to `255573`, its only `Maps to` points to itself, and in `concept_ancestor` it has exactly **one ancestor and one descendant: itself.** It is hierarchically isolated. So a grouping built from it does not come out wrong - it comes out **empty**, silently. That is not a quirk of this one concept: | Domain | Standard concepts | Hierarchically isolated | | -------------- | ----------------- | ----------------------- | | Condition | 167,045 | **0.1%** | | Procedure | 259,172 | 0.4% | | Drug | 2,022,915 | 2.0% | | Observation | 169,040 | 4.4% | | Measurement | 306,621 | 8.6% | | **Meas Value** | 45,908 | **89.7%** | `Meas Value` is where most cross-domain name collisions live (2,289 with `Observation`, 518 with `Condition`). Drifting into it costs you the hierarchy entirely. ### Fix it at resolution time Pass the domain you are populating. It is one parameter, supported on `/v1/search/semantic`, `/v1/search/concepts`, and `/v1/concepts/recommended`: ```bash theme={null} GET /v1/search/semantic?query=chronic+obstructive+pulmonary+disease &standard_concept=S&domain_ids=Condition ``` Every result is then in-domain and hierarchy-connected. Add a cheap guard after resolution: call `GET /v1/concepts/{concept_id}/level` and reject any concept whose `total_descendants` **and** `total_ancestors` are both `0`. That one check catches the entire isolated-concept class before it reaches your cohort logic. ### The one partial exception For morphology there is a usable recovery route. From a `Morph Abnormality` concept - which lives in `Observation` - `Asso morph of` reaches the conditions carrying that morphology: ```text theme={null} 4183159 Pulmonary blastoma (Observation / Morph Abnormality) --Asso morph of--> 40391740 Pulmonary blastoma (Condition / Disorder) 37167738 Primary pulmonary blastoma (Condition / Disorder) 36561198 Pulmonary blastoma of lower lobe, lung (ICDO Condition) ... 10+ more ``` Useful for oncology and ICD-O data, with two caveats: only **57.7%** of the 5,275 standard `Morph Abnormality` concepts have such a route, and it is **one-to-many**. You get a set of related conditions, not "the equivalent concept" - a fan-out, not a converter. ## 7. Putting It Together ```text theme={null} agent proposes terms │ ▼ resolve to seed concept_ids ◄── pass domain_ids; drop isolated concepts │ ▼ intersect /ancestors ──────► candidate groupers (coverage, then size) │ ▼ author expression {include, exclude} ◄── you version this │ ▼ expand → filter domain + standard_concept='S' │ ▼ concept_id list → WHERE condition_concept_id IN (...) │ ▼ /concepts/recommended ──────► anything outside the expansion? │ │ │ no - done │ yes - add as include seed ▼ └──────────► (back to expression) ``` Re-expand whenever you adopt a new vocabulary release. The expression stays the same; the concept list it produces gets better. ## 8. Scaling and Gotchas **Batch your lookups.** `POST /v1/concepts/hierarchy/batch` takes up to 100 queries in a single request and counts as **one** call against your rate limit (2 req/s on Free). Expanding 500 groupers is five requests, not five hundred. ```json theme={null} { "queries": [ { "query_id": "a", "concept_id": 4115260, "operation": "descendants", "params": { "max_results": 5000 } } ] } ``` **Set `max_results` explicitly on very large groupers.** The ceiling is 5,000 descendants per concept. If a grouper is broad enough to approach that, split it into several narrower `include` seeds rather than relying on one enormous root. **Set `max_levels=20` explicitly, always.** It defaults to `10`, and `truncated` does **not** flag depth truncation - only the row cap. A grouper whose tree runs deeper than 10 levels silently loses its deepest descendants while the response still reports `truncated: false`. This is the easiest way to build a quietly incomplete cohort. (A handful of SNOMED trees run deeper than 20, which is the API maximum; for those, seed from a lower node in the tree.) **Check the true descendant count before you expand.** `GET /v1/concepts/{concept_id}/level` returns `total_descendants` without fetching the set - a cheap way to catch a grouper that is far broader than you intended. **`include_invalid` defaults to `false`.** Deprecated concepts are excluded unless you ask for them. That is almost always what you want for a cohort; be deliberate if you override it. ## 9. Knowing When You Are Done The obvious worry about all of this is that you never had the exhaustive list of terms to begin with. You do not need one - and you could not verify it if you had it. **The grouper stabilizes long before the concept list does.** The two seeds in section 3 and the five-seed set both select `4115260`, with the same ranking. Three or four clinically diverse seeds is typically enough to fix the altitude. So replace "have I listed everything?" - unanswerable - with a **convergence test**: does my grouping already contain the things I have not thought of? Run `find_groupers` and pick the candidate just before the size cliff. Feed the same seeds to `POST /v1/concepts/recommended`. It implements the OHDSI **Phoebe** algorithm, suggesting concepts that co-occur with yours in real, curated concept sets. Run `GET /v1/search/semantic` over the same scope as a second, independent channel - it catches concepts worded differently from your extracted terms, where Phoebe's co-occurrence signal is thin. Check each suggestion against your expansion. A high hit rate means the grouper is at the right altitude. For the five lung seeds, filtered to `domain_ids: ["Condition"]` and `standard_only: true`: **71 distinct recommendations, 64 of them (90%) already inside `4115260`.** The ones that fall outside are your entire review queue - seven, in this case, each a real decision. `4154909` Abnormal radiologic density and `4027562` Radiologic infiltrate are genuine misses (in scope, but filed under the radiology branch rather than the anatomy branch). `256449` Bronchiectasis, `4050884` Pleural plaque and `4061819` Aortopulmonary window are judgement calls - airway, pleura, and vessel rather than lung parenchyma. One is co-occurrence noise. Add the genuine misses as new `include` seeds, re-expand, re-run. **Stop when the outside-list contains only things you are deliberately rejecting.** That is your termination condition. Use at least two or three seeds, and make them clinically diverse. A single seed carries no information: every one of its ancestors "covers" it, so the tightest candidate is just its immediate parent. The intersection only becomes informative when the seeds disagree about where they sit. `POST /v1/concepts/recommended` returns `data` as an object **keyed by source concept ID**, not a flat list. A concept suggested by three seeds appears three times, so de-duplicate on `concept_id` before you count - the 84 rows in the lung example are 71 distinct concepts. The expression grows by a line or two per round. The maintenance burden does not.