# Correlation Studio — User Guide This guide describes every user-facing feature, page, control, and option in Correlation Studio. It is the source of truth for **Corrie**, the in-app help assistant. Each section is written to be self-contained so it can be retrieved and answered without context from surrounding sections. Terminology used throughout this guide matches the labels shown in the application UI: **Dataset**, **Experiment**, **Discovery**, **Portfolio**, **Post**, **Message**, **Workgroup**, **User**. --- ## 0. Corrie — the in-app assistant Corrie is the chat assistant built into Correlation Studio. The name is a play on "correlation" — she's the platform's helper for navigating the product and the data published on it. Open Corrie from the floating chat button in the bottom-right corner of any signed-in page. Ask questions about how the product works ("how do I publish a portfolio?", "what does Granger causality mean here?") or about specific data published on the site ("what correlations have been published on climate?"). Corrie answers from two sources: this user guide (the Docs corpus) and the public Discoveries / Experiments / Portfolios / Datasets / Forum Questions that users have published (the Entity corpus). Each answer includes citations linking back to the source so you can verify. Corrie does not have access to private content — only public published entities and this documentation are indexed. Anonymous visitors do not see Corrie; sign in first. Chat history is saved per-user; open a prior conversation from the chat dropdown, or start a new one with the **+** button. Each query costs tokens at the configured per-query rate (visible on the Usage page); the panel will warn you if your balance is empty. --- ## 1. What Correlation Studio Is Correlation Studio is a web application for finding statistical correlations between datasets. A user uploads or imports two datasets, pairs them in an **Experiment**, and the system computes a Pearson correlation coefficient for every compatible column pair. Each column-pair result is a **Discovery**, and the strongest discoveries can be published, analyzed with AI, organized into **Portfolios**, and discussed in **Posts**. The application is organized around five primary entities: - **Dataset** — a tabular table with named, typed columns and rows of values. - **Experiment** — a pairing of two datasets (called the X dataset and the Y dataset) plus parameters that control how their rows are joined and how strict the correlation threshold is. - **Discovery** — the result of correlating one column from the X dataset against one column from the Y dataset. Every experiment produces many discoveries, one per compatible column pair. - **Portfolio** — an authored composition of markup, images, audio/video embeds, and references to datasets, experiments, and discoveries. Portfolios are how findings are written up and shared. - **Post** — a community discussion thread in the public feed. Posts can attach images, audio/video, and references to public datasets, experiments, discoveries, and portfolios. Two additional entities support collaboration: - **Workgroup** — a named group of users used to share entities privately with a defined audience that is wider than just the owner but narrower than the public. - **Message** — direct user-to-user messaging with the same attachment model as Posts. --- ## 2. Accounts and Roles ### Roles Every user has exactly one role. Roles are hierarchical: a higher role inherits all permissions of lower roles. - **Anonymous** — not signed in. Can browse public discoveries on the Home feed and view public entity pages. Cannot create, comment, rate, message, or perform any write action. - **Registered** — signed-in standard user. Can create and own Datasets, Experiments, Discoveries, Portfolios, Posts, and Messages. Can comment, rate, tag, set sentiment, and join Workgroups. - **Moderator** — Registered powers plus the ability to delete comments and posts created by other users. - **Curator** — Registered powers plus the ability to manage topics in the Dataset Catalog. - **Administrator** — full access including the Administration page and all user/billing/system-management features. ### Registration The **Register** page is reached from the top-right of any page when signed out. The form asks for username, email address, password, first name, and last name. After registering, the user receives a verification email. The account is usable immediately but is marked unverified until the link in the verification email is clicked. ### Login The **Login** page accepts username (or email) and password. Successful login redirects to the Home page. The page also offers an external login option for supported identity providers and a link to recover a forgotten password. ### Password recovery The **Forgot password** link on the Login page accepts the user's email address and sends a recovery email containing a reset link. The link expires after a short window. ### Sign out The user menu in the top-right header includes a **Sign out** option that ends the current session and returns to the Home page. ### Tokens and subscriptions **Tokens** are the unit of cost for billable actions in Correlation Studio. The current balance is shown in the header next to the user menu. The exact token cost of any action is displayed before the action runs and again after it completes; the per-action history is available on the Usage page. Token-billed actions fall into three categories: - **AI actions** — every AI analysis (dataset, experiment, discovery, portfolio) and every AI dataset search (Remote Search mode in the Dataset wizard, querying Claude, Gemini, and Grok). The cost reflects the AI provider's input and output token pricing plus any grounding events the provider bills for separately. - **Ingestion actions** — uploading and ingesting a Dataset consumes tokens proportional to the dataset's file size, row count, and column count. Running an Experiment also consumes tokens, scaled by the number of column pairs to be correlated (X columns × Y columns) and the row count of the joined data. Both Dataset ingestion and Experiment ingestion show their estimated token cost in a confirmation prompt before the action starts. - **Tooling actions** — utility operations like rendering a discovery thumbnail on demand, refreshing a remote dataset's source file, or running a portfolio's Analysis block. Each tool's exact cost is shown next to its action button. If any of these actions fails after tokens have been charged — ingestion error, AI provider error, network failure — the tokens are refunded automatically and the refund appears as a credit in the Usage page's per-action breakdown. Subscriptions provide a monthly token allowance plus optional one-time top-ups. The active subscription is visible on the **Profile** page under the **Subscriptions** section. Subscriptions can be upgraded, downgraded, or canceled from that section. Tokens included in the monthly allowance roll over according to the plan's rollover policy; purchased top-up tokens do not expire. --- ## 3. Visibility States Every Dataset, Experiment, Discovery, and Portfolio is in exactly one of three visibility states. The state is called the **published state** in the UI. - **Private** — visible only to the owner. The default state for newly created entities. Does not appear in the Home feed, the Browse pages, or any search results other than the owner's own "My" lists. - **Public** — visible to every user, including anonymous visitors. Appears in the Home feed, search results, and topic catalog if categorized. Other users can comment, rate, tag, and set sentiment on Public entities. - **Workgroup** — visible to members of a specified workgroup plus the owner. Does not appear in the public Home feed or anonymous search. Workgroup members see the entity in their own dashboards. The visibility state of an entity is changed using the **Publish** and **Unpublish** buttons on its detail page. A Workgroup share is configured through the **Share** button, which opens a picker for the target workgroup. Publishing a Dataset, Experiment, Discovery, or Portfolio with status Public makes it visible on the Home page feed and in Search results. Unpublishing removes it from those surfaces and returns it to Private. --- ## 4. Header and Navigation The persistent header bar at the top of every page contains the following elements, in left-to-right order: - **Application logo and name** — clicking returns to the Home page. - **Search bar** — a free-text search for public discoveries, experiments, datasets, portfolios, and posts by title or description. As you type (3+ characters) a dropdown shows the top quick-pick matches grouped by type; clicking one jumps straight to that item's detail page. Pressing **Enter** instead opens the full search-results view on the Home page (see *Searching from the Home page* below). - **Primary navigation links** — Home, Datasets, Experiments, Portfolios, Posts, Messages, Workgroups. Each link is highlighted when the user is on that section. - **Notification bell** — shows the unread notification count as a badge. Clicking the bell opens a panel listing recent notifications. - **Token balance** — the user's current available token count. Clicking opens the Usage page. - **User menu** — avatar plus username. Clicking opens a menu with links to Profile, Usage, and Sign out. Administrators also see an Administration link. The footer is a thin status strip that may show ingestion progress, server status, or other transient activity indicators. --- ## 5. Home Page The **Home** page is the published-content feed and the default landing page after login. It is reached at the root path `{{BaseUrl}}/`. ### Feed contents The Home feed is a layered stream of published content, arranged so there is always something new to explore. After a **Pinned** section at the very top (cards Curators/Administrators have pinned there, marked with a blue Pinned badge), the feed cycles through repeating rounds: 1. **Trending** — all of the newest day's **Trending: {topic}** bands together. A daily AI job (when enabled) identifies topics trending in world news, matches them against the public corpus, and renders each topic as a full-width band of matching cards. 2. **Featured** — a short strip of the most recently featured cards (amber Featured badge). 3. **Fresh from the community** — the newest publishes by community members (regular registered users), with a slim section heading. Staff publishes do not get this boost — the section exists to celebrate community activity. 4. **The mix** — a run of random public content, reshuffled once per day. Scrolling deeper repeats the cycle with the *previous* day's trending bands and *earlier* community publishes, and so on — until trending history and recent community activity run out, after which the feed is a pure daily-shuffled mix with no duplicates within a day. Tomorrow the whole mix is different. Only the newest few featured cards get the strip; older **Featured** content carries its amber badge wherever it appears in the mix, and the **Featured** filter chip lists it all. (Featured is the editorial quality flag; Pinned is deliberate top placement — a card can be both.) Every so often in the mix, an experiment expands into a **cluster** — a mini-portfolio showing the experiment card, chips linking to its X and Y datasets, and a scrollable row of its strongest discoveries as compact chart tiles. Each feed card shows: - Title, owner username, and creation date. - Sparkline or scatterplot preview of the underlying chart (or a data preview for datasets). - Correlation coefficient `r` for discoveries, mean `r` for experiments. - Rating and thumbs-up/down sentiment summary. A card is clicked to open the corresponding detail page. ### Featured and Pinned content Two Curator/Administrator mechanisms promote content on the Home feed — regular users cannot push their own content to the top: - **Pinned** holds a card at the very top of the feed. Pin from the card's **⋯** menu on the Home feed, or with the pin button next to the Home-feed toggle on any entity's detail page. - **Featured** is the editorial quality flag: an amber badge wherever the card appears, and inclusion in the **Featured** filter chip. A card can be both Pinned and Featured. User-level pinning is a separate mechanism that does NOT affect the Home page. The **Pin to Profile** action pins a user's own post to the top of their public profile page; see the Posts section for details. There is no user-level pin that targets the Home feed. ### Sorting and filtering Filter chips at the top of the feed limit results to a specific entity type (All, Featured, Posts, Portfolios, Experiments, Discoveries, Datasets). Choosing a type shows just that type of content, shuffled daily like the main mix (no trending bands or clusters in a filtered view). The **Featured** chip lists all featured content newest first. ### Moderating from the feed Card owners and Curators/Administrators see a small **⋯** menu on feed cards with quick moderation actions: - **Hide from feed** — removes the card from the Home feed without unpublishing. The content stays Public, searchable, and reachable by link; it just stops appearing as a feed card. Available to the content's owner and to Curators. - **Unpublish** — makes the content Private (removed from Home, search, and public pages), after a confirmation. The owner can republish from the detail page. - **Feature / Unfeature** — Curators/Administrators only; toggles the Featured flag. - **Pin to top / Unpin from top** — Curators/Administrators only; holds the card at the very top of the feed, above Featured. - **Remove from band** — Curators/Administrators only, on cards inside a trending band; drops that card from the band. A band that gets too small hides itself. Curators can also hide an entire trending band with the **×** button in the band's header. ### Searching from the Home page To search Correlation Studio, type a term in the header search bar and press **Enter**. This filters the Home page to show every matching piece of published content rendered as the same feed panels the normal Home feed uses (charts, mini-charts, dataset previews, thumbnails) — not just a dropdown list. Searching matches public discoveries, experiments, datasets, portfolios, and posts by their **title or description**. Multi-word queries match on **any** word and rank items containing all of them (and exact in-order phrases) highest; wrap text in `"double quotes"` for an exact-phrase match. See [Search](#16-search) for the full matching rules. While in the search-results view: - The same entity-type **filter chips** (All / Discoveries / Experiments / Datasets / Portfolios / Posts) narrow the results to a single type. For example, searching `climate` and clicking **Discoveries** shows only matching discoveries. - A header strip shows the active search term and the total number of matches, with a **Clear** button to return to the normal Home feed. - Results are grouped by type (discoveries first, then experiments, datasets, portfolios, posts) and load more as you scroll. - Only **Public** content appears (private and workgroup-only content is excluded). **Searching by URL.** The search view is fully addressable by URL, so a search can be linked or bookmarked: - `{{BaseUrl}}/?q=climate` — opens the Home page filtered to everything matching "climate". - `{{BaseUrl}}/?q=climate&entityTypeId=3` — the same search narrowed to a single entity type. The `entityTypeId` values are: Dataset = 1, Experiment = 2, Discovery = 3, Portfolio = 4, Post = 5. **Year search.** A bare 4-digit year (e.g. `1970`) searches discoveries whose underlying data covers that year (using each discovery's date-range), rather than a text match — useful for finding time-series correlations from a specific period. --- ## 6. Datasets ### What a Dataset is A **Dataset** is a tabular dataset with named columns and rows of values. Every dataset belongs to one user (the owner). A dataset has a title, optional description, a published state, an optional source file or remote source URL, and one or more columns. The minimum useful dataset has two columns and at least three rows, and at least one column must be numeric (Integer or Decimal) so that correlation analysis has something to compute. ### Datasets list page The **Datasets** page (`{{BaseUrl}}/datasets`) shows the user's datasets and public datasets in two tabs. - **My Datasets** tab — every dataset owned by the signed-in user, regardless of published state. Each row shows title, file type icon, dataset state, row count, column count, published state badge, and tags. - **Public Datasets** tab — every dataset with published state Public, owned by any user. Same columns as My Datasets. Each row in the table has: - A drag handle (the gripper icon at the far left). Dragging a row up or down changes the dataset's sort order and persists the new order across sessions. - A title link that opens the dataset detail page. - A **Publish** / **Unpublish** button (only on the user's own rows). - A **Delete** button (only on the user's own rows). Deletion asks for confirmation and removes the dataset, its rows, and any experiments that depend on it. Above the table is a **+ New Dataset** button that opens the Dataset wizard. A search box, topic filter, and a **Modified-date filter** (All time / Today / This Week / This Month / Last Month / Last 6 Months / Last Year / Custom) narrow the list; the date filter matches each dataset's last-modified date, is saved in the page URL, and is respected by "Select all N matching" bulk actions. #### Bulk actions Each row in the **My Datasets** tab has a checkbox. Ticking one or more reveals a **bulk-action toolbar** above the table that applies a single action to the whole selection in one server request: - **Make Public** / **Make Private** — change the published state of every selected dataset. - **Publish** / **Unpublish** — add or remove the selected datasets from the Home feed (only Public datasets can be on the feed). - **Analyze** — queue background AI analysis for every selected dataset. The toolbar shows an estimated token cost and your balance before you confirm; the work runs server-side and the results appear as each one finishes. - **Delete** — delete every selected dataset (with confirmation). Two layers of "select all" let bulk actions reach beyond what's on screen: - The **header checkbox** selects every row on the current page. - When every row on the page is selected and more rows match the current filter, a banner offers **Select all N matching the current filter**. This extends the action to the entire result set — not just the loaded page — so you can act on hundreds or thousands of rows at once. The matching set is resolved on the server, so it stays correct no matter how many pages there are. A **Clear selection** link cancels the all-matching mode. ### Dataset wizard The Dataset wizard creates a new dataset. It is opened at `{{BaseUrl}}/datasets/new` from the **+ New Dataset** button on the Datasets page. The wizard has five steps: 1. **Source** — pick the source. Required field: **Description** (free text, used by the AI search providers when in Remote Search mode and as the dataset's description otherwise). The Import Source choice is one of: - **Local Files** — upload one or more CSV, TSV, Excel, or HTML files from the user's computer. - **Web Links** — provide one or more URLs to download. - **Remote Search** — describe what the user is looking for and let the AI providers find dataset URLs. Local Files mode also offers a **Skip the line. Paste & go.** checkbox that switches to pasted-content mode. In pasted-content mode the wizard accepts CSV or TSV text pasted into a textarea and skips the file upload step. Remote Search mode shows additional optional fields: - **Date Range** — free text like "2020 to 2024" or "last 5 years". - **Geography** — free text like "United States", "Global", "Europe". The Remote Search mode displays the live status of each provider (Claude, Gemini, Grok) as the AI search runs, with a count of returned and rejected URLs per provider, plus a panel listing every rejected URL and the rejection reason (HTTP 404, unsupported file extension, HTML page returned instead of data, etc.). 2. **Results** (Remote Search mode only) — a list of dataset candidate URLs returned by the AI providers. Each candidate has a checkbox; checked candidates proceed to download. 3. **Download** — the selected files are downloaded in parallel. Each file shows a progress bar and a status (downloading, downloaded, failed). Failed downloads can be retried with the **Retry** button on the row. 4. **Columns** — the user reviews the detected columns for each file. Files with no recognized columns or no numeric columns are pruned automatically and listed in a "Pruned files" panel with the reason. For each remaining file the user can: - Edit a column's **Label** (the display name). - Change a column's **Datatype** (Integer, Decimal, Date, DateTime, Text, Geo). - Toggle **Selected** to include or exclude the column from correlation analysis. - Toggle **Join Key** to mark the column as a join key for SharedKey or TimeSeries joins. - Set the column's **GeoType** — the geographic identifier a column holds (Latitude, Longitude, City, State, Zipcode, County, **FIPS**, **ZCTA**, Country, and more). GeoType drives the Geographic map and the **Geo Type** experiment join. Common geographies are pre-filled automatically from the column name — including **FIPS** codes (labels like `fips` / `geoid` / `county_fips`, or any integer "county" column — FIPS codes are always numeric, so a numeric county column is treated as a code rather than a name) and **ZCTA** codes (a `zcta` label on an integer column). You can change any of it here. - Add an optional **Hint** describing what the column represents. The per-column **Aggregate Function** (how duplicate keys are combined during a Shared Key or Geo Type join — Sum, Average, Min, Max, First, Last, Count) is not shown in the wizard; it defaults to **Average** and can be changed later on the **dataset detail page's Columns tab**. 5. **Preview** — a final review showing the dataset's title, description, column list, and a sample of the first rows. The **Create** button commits the dataset; the **Back** button returns to a prior step. The Next button on every step is disabled until the step's required inputs are valid. When advancing from step 1 the Next button shows a spinner during the brief topic-creation and file-pruning work that happens before the next step renders. ### Dataset detail page The **Dataset detail** page (`/datasets/{publicId}`) shows everything about a single dataset. The layout has: - A header with title, owner, published state badge, creation date, last modified date, and file size. - An **Edit title** and **Edit description** button (owner only). - A **Tags** row showing all tags on the dataset; the owner can add or remove tags. - A **Comments** section listing every comment with author, time, and reply count. The signed-in user can post a new comment or reply to an existing one. - A **Rating** widget showing the average rating and a per-user rating input (1 to 5 stars). - A **Sentiment** widget showing the count of Positive, Neutral, and Negative reactions, with controls to set the current user's sentiment. - A **Columns** table listing every column with its label, datatype, source index, selected flag, join key flag, geo data type, aggregate function, and hint. Columns can be reordered by drag-and-drop; their selected, join key, datatype, geo type, aggregate, and hint values can be edited inline. Columns created with the computed-column editor show a **ƒ** badge next to their label — hover it to see the expression they were computed from. The owner of a Ready dataset also sees an **Add computed column** button above the table (see [Computed columns](#computed-columns) below). - A **Distributions** section showing one histogram or box plot per numeric column. - A **Correlation preview** section showing column-pair correlations within the same dataset (useful for identifying redundant columns). - A **Rows** preview showing a sample of rows (paginated). - Action buttons in the header: - **Publish** / **Unpublish** — toggle the dataset between Private and Public. - **Share** — open the Workgroup picker to share with a workgroup. - **Ingest** — run or re-run ingestion (parses the source file and populates rows). Required after any column type change. - **Refresh File** — re-download from the remote source (Remote Search and Web Link datasets only). - **Analyze** — request an AI analysis of the dataset. Costs tokens. - **Delete** — permanently delete the dataset and all dependent experiments. Requires confirmation. Once a dataset is published, anyone — including signed-out visitors — can export it to CSV with the **Export** button, and it becomes citable as open data (see [Open Data and Exports](#18-open-data-and-exports)). ### Computed columns A **computed column** is a new column whose values are calculated from an expression over the dataset's existing columns — for example a price with tax, a ratio of two columns, a cleaned-up text field, or a pass/fail flag. The owner of a **Ready** dataset adds one from the **Columns** tab via the **Add computed column** button. The editor has: - A **Column name** field for the new column's label (must not duplicate an existing column). - An **Expression** box. Reference columns in square brackets — `[Column Name]` — write text in single quotes (`'like this'`), and join text with `&`. Numbers, `true`, `false`, and `null` are literals. Arithmetic (`+ - * / % ^`), comparisons (`= != < <= > >=`), and `and` / `or` / `not` all work. - A **column picker** — click any column chip to insert its reference at the cursor. - A **function library** organized by category — **Math** (ABS, ROUND, FLOOR, CEIL, SQRT, POW, EXP, LN, LOG10, MOD, SIGN, MIN, MAX), **Text** (UPPER, LOWER, TRIM, LEN, LEFT, RIGHT, SUBSTR, REPLACE, CONCAT, CONTAINS, STARTSWITH, ENDSWITH, LPAD, RPAD, SPLITPART), **Date** (YEAR, MONTH, DAY, HOUR, MINUTE, WEEKDAY, WEEK, QUARTER, DATEDIFF, TODAY), **Logic** (IF, COALESCE, ISNULL, NULLIF), and **Convert** (NUMBER, INT, TEXT, DATE). Click a function to insert it; hover for its signature, description, and an example. - A **Validate & preview** button that checks the expression and evaluates it against the dataset's first 10 rows, showing the referenced columns side-by-side with the computed result and the inferred column type. **Add column** only enables after a successful preview of the current expression. Example expressions: - `ROUND([Price] * 1.08, 2)` — price with 8% tax, 2 decimals. - `[Revenue] / [Units]` — per-unit revenue (division by zero yields an empty value, not an error). - `IF([Score] >= 60, 'Pass', 'Fail')` — a categorical flag; `IF([Score] >= 60, 1, 0)` produces a numeric 0/1 column suitable for correlation. - `UPPER(TRIM([State]))` — normalized text for joining. - `DATEDIFF('day', [Listed], TODAY())` — days since a date column. - `[City] & ', ' & [State]` — concatenated text. Behavior notes: - The column is **computed once** over every row when you click Add — the dataset briefly shows Queued/Reading while the values are materialized, then returns to Ready. It then behaves like any other column, including in experiments and the Distributions/Quality tabs. - **Messy data degrades gracefully** — a value that can't be interpreted for the operation (e.g. text where a number is needed) produces an empty value for that row rather than failing the whole computation. Writing `[Column] = null` checks for empty values. - Adding a computed column is billed like a small ingestion (a base cost plus the storage the new column actually adds), and the added storage counts toward your quota. - Up to 5 computed columns can be queued per request; the editor adds one at a time. - **Re-ingesting the dataset from its source file removes computed columns** — the data is rebuilt from the file, which doesn't contain them. Re-add them after a re-ingest if needed. ### Dataset states Every dataset has a state that reflects its position in the ingestion pipeline. The state badge on the dataset detail page and on every list shows one of: - **Pending** — created but not yet queued for ingestion. - **Queued** — waiting in line for an ingestion worker. - **Reading** — the worker is parsing the source file. - **Read** — file parsed, rows are being staged. - **Waiting** — waiting for the indexing phase. - **Indexing** — rows are being indexed for correlation lookup. - **Ready** — fully ingested and ready to use in experiments. - **Failed** — ingestion failed. The dataset detail page shows the failure reason. Tokens charged for ingestion are refunded automatically when a dataset reaches Failed state. A dataset must be in the Ready state before it can be used in an experiment. ### Dataset Catalog The **Dataset Catalog** page (`{{BaseUrl}}/catalog`) is a topic-organized browser for public datasets. The page has a tree of topics on the left and a list of datasets in the selected topic on the right. - **Topics** are categories like "Climate", "Finance", "Public Health". Topics can have sub-topics, forming an ontology. - **Curators** and **Administrators** can create, rename, merge, reorder, and delete topics. - A dataset is added to a topic by the owner from the dataset detail page, or by a curator via the **Add Dataset to Topic** button on the catalog page. - A dataset can belong to multiple topics. --- ## 7. Experiments ### What an Experiment is An **Experiment** pairs two datasets (the X dataset and the Y dataset) and configures how their rows are joined and how strict the correlation threshold is. Running ingestion on the experiment produces one **Discovery** per compatible column pair, where each discovery carries a Pearson correlation coefficient `r` between roughly -1 and +1. An experiment has a title, optional description, an absolute correlation threshold, a join type, an optional join tolerance, a default visualization type, an owner, a published state, and a state that tracks ingestion progress. ### Experiments list page The **Experiments** page (`{{BaseUrl}}/experiments`) shows the user's experiments. Each row shows title, X and Y dataset names, discovery count, mean correlation coefficient, ingestion state, published state, and tags. Each row has: - A drag handle for reordering. - An expand toggle (chevron) that reveals the experiment's discoveries as nested sub-rows. The sub-rows can themselves be drag-reordered. - A title link that opens the experiment detail page. - Publish / Unpublish, Share, and Delete actions. The **+ New Experiment** button at the top of the page opens the Experiment create page. A search box and a **Modified-date filter** (All time / Today / This Week / This Month / Last Month / Last 6 Months / Last Year / Custom) narrow the list; the date filter matches each experiment's last-modified date, is saved in the page URL, and is respected by "Select all N matching" bulk actions. #### Bulk actions Each experiment row has a checkbox that reveals the same style of **bulk-action toolbar** described under [Datasets → Bulk actions](#bulk-actions): **Make Public**, **Make Private**, **Publish** / **Unpublish** (to/from the Home feed), **Analyze** (queues AI analysis for every child discovery of the selected experiments), and **Delete**. The two-layer **select all on this page** / **Select all N matching the current filter** banner works the same way, so a bulk action can target every experiment that matches your filter, not just the visible page. ### Creating an experiment The **Experiment create** page (`{{BaseUrl}}/experiments/new`) has the following fields: - **Title** — required. The display name. - **Description** — optional free text. - **X Dataset** — required. A picker listing the user's datasets and any public dataset they have access to. The picker shows each dataset's title, column count, and row count. - **Y Dataset** — required. Same picker as X Dataset; can be the same dataset as X (self-correlation). - **Absolute Correlation Threshold** — a slider or numeric input from 0 to 1. Only column pairs whose absolute correlation `|r|` meets or exceeds this threshold produce a discovery. Higher values are stricter and produce fewer, stronger discoveries. - **Join Type** — one of: - **Row Sequence** — pairs rows by their position in the source file (row 1 of X with row 1 of Y, etc.). The simplest join; both datasets must have the same number of rows. - **Shared Key** — pairs rows by a shared key column. Both datasets must have a column marked as Join Key, and the key columns must contain matching values. - **Time Series** — pairs rows by closest timestamp. Both datasets must have a Date or DateTime column marked as Join Key. The Join Tolerance setting controls how far apart two timestamps can be and still be paired. - **Geo Type** — pairs rows by shared geography. Both datasets need a column tagged with the same **GeoType** (set on the dataset's Columns tab, often auto-detected from the column name): the most specific type both share is used — Zipcode (or **ZCTA**, which is treated as the same thing), then City, **FIPS**, County, State, Province, Country, Continent — or, when both datasets carry a Latitude *and* Longitude column, the coordinate pair itself (matched at 4-decimal precision, roughly 11 meters). A **ZCTA** (ZIP Code Tabulation Area) column and a **Zipcode** column join each other, since they describe the same 5-digit geography. Values are matched case-insensitively; zip/ZCTA codes normalize to their 5-digit form, and FIPS / census GEOID codes normalize with their leading zeros restored (a canonical code joins far more reliably than county names). Many rows typically share one location — those rows are combined per column using each column's Aggregate Function (Average by default), exactly like duplicate Shared Key values. - **Join Tolerance Seconds** — only shown for Time Series joins. The maximum allowed gap between two paired timestamps, in seconds. - **Default Visualization Type** — one of: - **Scatterplot** — X column on the horizontal axis, Y column on the vertical axis, one dot per paired row. - **Line Graph** — both columns plotted as lines along a shared horizontal axis. - **Residual Plot** — scatterplot of residuals after fitting a linear regression line. - **Tags** — optional list of free-text tags. The page also has three ingestion-filter checkboxes that control which column pairs become discoveries: - **Ignore identity discoveries** (default on) — skips pairs whose correlation is exactly `r = 1.0` (a column against itself, a perfect-duplicate column, or a perfect linear transform). These are analytically trivial and would just be noise. - **Ignore duplicate discoveries** (default on) — skips any column pair you already have a discovery for in another experiment. This is the recommended setting when you grow a **dataset corpus over time**: re-running the same cross-matrix experiment as you add datasets would otherwise re-compute every pair you already have. With this on, only the genuinely new pairs (those involving a newly added dataset) produce discoveries — which also saves ingestion time and tokens, since duplicate pairs are skipped *before* the correlation is computed. A pair counts as a duplicate when both its X column and Y column exactly match an existing discovery's, in the same orientation. - **Ignore experiments with no results** (default on for multi-dataset runs) — if an experiment ends up with zero qualifying discoveries (all pairs fell below the threshold or were skipped), the experiment is auto-deleted instead of left empty. The **Create** button validates the form, creates the experiment in Pending state, and starts ingestion automatically. ### Experiment detail page The **Experiment detail** page (`/experiments/{publicId}`) shows the experiment header, parameters, ingestion progress, and a navigator over its discoveries. The page has the following sections: - **Header** — title, owner, published state badge, X dataset link, Y dataset link, join type, threshold, mean correlation, discovery count, ingestion state, and a progress bar if ingestion is in progress. - **Action buttons** — Edit, Publish/Unpublish, Share, Ingest (re-run ingestion), Cancel (running ingestion only), Analyze (AI analysis), Delete. - **Discoveries navigator** — a list of every discovery with column pair names and correlation coefficient. The navigator supports multiple view modes selected by the toolbar above it: - **List view** — text rows; click to open the discovery detail page in a side panel. - **Sparkline view** — each discovery shown as a small inline chart preview. Sparkline thumbnails are rendered server-side automatically for visible discoveries that don't yet have one. - **Heatmap view** — a grid of column pairs colored by correlation strength. Hovering shows the column names and `r` value. - **Bubble view** — discoveries plotted as bubbles where bubble size encodes the row count and color encodes correlation strength. - **Divergence view** — pairs of column means visualized as a divergence chart, useful when one dataset is a baseline and the other is a comparison. - **Lag view** — for time-series joins, the correlation coefficient computed at multiple lag values, showing leading and lagging relationships. - **Network view** — discoveries rendered as a graph with columns as nodes and edges weighted by correlation strength. - **Tags, comments, rating, and sentiment** — same as on the dataset detail page. - **AI analysis** — present after the experiment owner runs an analysis; shows the AI-generated summary of the experiment's findings. ### Experiment states The experiment state badge shows one of: - **Pending** — created but not yet queued. - **Queued** — waiting for an ingestion worker. - **Running** — ingestion is currently computing correlations. - **Ready** — ingestion is complete; all discoveries are populated. - **Failed** — ingestion failed. The experiment header shows the failure reason. Tokens charged for ingestion are refunded automatically. A user can **Cancel** a Queued or Running experiment from the detail page. Canceled experiments return to Pending state; the user can edit parameters and re-run ingestion. ### Reordering discoveries On the experiment detail page list view, the owner can drag-reorder discoveries. The new order is persisted and affects the order on the experiment's published page and in the discoveries navigator. ### Bulk actions on discoveries In **List view**, each discovery row has a checkbox. Ticking one or more reveals a **bulk-action toolbar** above the list — the same shape as the bulk toolbars on the Datasets and Experiments list pages: - **Make Public** / **Make Private** — change the published state of every selected discovery. (Make Public only appears when the parent experiment is itself public, since a discovery can't be public under a private experiment. Any individual discovery that can't be flipped — for example one whose parent is still private — is reported as a skipped count rather than failing the whole batch.) - **Publish** / **Unpublish** — add or remove the selected discoveries from the Home feed (only Public discoveries can be on the feed). - **Analyze** — queue background AI analysis for every selected discovery. The confirm dialog shows the estimated token cost against your balance; the Brain badge lights up on each row as the server finishes it. - **Delete** — delete every selected discovery (with confirmation). The discovery navigator loads discoveries in pages as you scroll, so the toolbar offers two layers of "select all" to reach a large experiment's full set: - The **header checkbox** selects every discovery currently loaded. - When every loaded discovery is selected and more match behind the scroll, a banner offers **Select all N matching** (it respects the discovery title search if one is active). This extends the action to *every* discovery in the experiment, resolved on the server, so a bulk Publish / Analyze / Delete can act on thousands of discoveries without scrolling each page into view. A **Clear selection** link cancels it. This is the recommended way to publish all of a large experiment's discoveries at once: open the experiment, switch to List view, tick the header checkbox, choose **Select all N matching**, then **Make Public**. ### Publishing an experiment The **Publish** button on the experiment detail page changes the experiment's published state to Public. Publishing an experiment automatically generates thumbnail images for every discovery so they render quickly on the Home feed. The button hint reads "Publish to Home & Search". The opposite button reads "Unpublish from Home & Search". --- ## 8. Discoveries ### What a Discovery is A **Discovery** is one row of correlation analysis: column `i` of the X dataset correlated against column `j` of the Y dataset, with a Pearson correlation coefficient `r` and the underlying paired data points. Every discovery belongs to exactly one experiment. A discovery has a title (generated from the column names), an `r` value, a join type, a visualization type, an owner, a published state, optional analysis text, and a thumbnail. ### The correlation coefficient: `r` and `|r|` Correlation Studio uses two closely related numbers, and it is important to know which is which because they appear in different places in the UI. - **`r` (the signed Pearson correlation coefficient)** is the actual correlation between the two columns. It ranges from **-1 to +1**. The **sign** carries meaning: a positive `r` means the two columns move in the same direction (when one goes up, the other tends to go up); a negative `r` means they move in opposite directions (when one goes up, the other tends to go down); an `r` near 0 means little to no linear relationship. The closer `|r|` is to 1, the stronger the relationship. - **`|r|` (absolute value of `r`)** is the **strength** of the correlation regardless of direction. It ranges from **0 to 1**. A strong negative correlation (`r = -0.92`) and a strong positive correlation (`r = +0.92`) have the same strength (`|r| = 0.92`). **Where signed `r` is used:** - The `r` value shown on each **Discovery** card, the Discovery detail page header, and the Home feed cards — always the signed value, so you can tell direction at a glance. - The **color coding** of correlation values: positive correlations and negative correlations are colored differently, and the intensity reflects the magnitude. - The **Discoveries navigator** on the experiment detail page, and the **charts** (scatterplots, line graphs, sparklines, mini-charts). **Where absolute `|r|` is used:** - The experiment's **Absolute Correlation Threshold**. During ingestion, a column pair becomes a published Discovery only if its `|r|` meets or exceeds this threshold. This is why a strong *negative* correlation still passes a high threshold — the gate is on strength, not sign. - **Ranking and sorting discoveries by strength**: the strongest discoveries (highest `|r|`, positive or negative) are surfaced first in default orderings, search results, and the top-N selections used for experiment thumbnails and mini-charts. - The experiment's **mean correlation coefficient** (shown on the Experiments list and experiment header) is the **mean of `|r|`** across all of the experiment's discoveries. It is therefore always non-negative — it summarizes the average *strength* of the experiment's correlations, not their average direction. In short: read **`r`** to learn the direction and strength of a single correlation; the system uses **`|r|`** wherever it needs to compare or threshold correlations by strength alone. ### Discovery detail page The **Discovery detail** page (`/discoveries/{publicId}`) shows the discovery's chart, drilldown tables, and additional analysis views. The page is reached from the Home feed, from the experiment detail page navigator, or directly via URL. The page has: - A **Header** with title, parent experiment link, both column names, `r` value, owner, and published state badge. - A **Primary chart** rendered in the visualization type set on the discovery (Scatterplot, Line Graph, or Residual Plot). The chart can be re-rendered in any of the three types using the **Visualization** toggle above it. - A **Drilldown tables** section showing the underlying X and Y values paired row-by-row with their join keys. - A **Regression** section showing the linear regression line fit, the slope, intercept, R-squared, and the equation in formula form. - A **Lag analysis** chart (time-series joins only) showing how `r` changes as the Y series is shifted forward or backward in time. - A **Rolling correlation** chart showing how `r` varies over a rolling window across the joined timeline. - **Tags, comments, rating, and sentiment** — same controls as other detail pages. - **AI analysis** — if requested by the owner, shows the AI-generated interpretation of the correlation. - **Action buttons** — Edit, Publish/Unpublish, Share, Ingest (re-run with a different join type or threshold), Analyze, Render Thumbnail, Delete. ### Visualization types Three visualization types are available on every discovery: - **Scatterplot** — each paired data point is a dot at (x value, y value). Best for static correlations. - **Line Graph** — X and Y values plotted as lines along the same horizontal axis (row index or time). Best for time-series. - **Residual Plot** — scatterplot of residuals (actual Y minus predicted Y from the regression line). Best for spotting non-linear patterns or heteroscedasticity. The default visualization type is inherited from the parent experiment but can be overridden per-discovery from the discovery detail page. Changing the visualization type triggers a thumbnail re-render so that the Home feed preview matches the chart. ### Thumbnails Every published discovery has a thumbnail image used on the Home feed and in sparkline views. Thumbnails are generated server-side when the discovery is first published, when its visualization type changes, and on demand via the **Render Thumbnail** button on the detail page. --- ## 9. Portfolios ### What a Portfolio is A **Portfolio** is an authored, block-based composition that combines markup, images, audio/video embeds, and references to datasets, experiments, and discoveries. Portfolios are how findings are written up for publication. A portfolio has a title, optional description, an owner, a published state, and an ordered list of blocks. ### Portfolios list page The **Portfolios** page (`{{BaseUrl}}/portfolios`) shows the user's portfolios in a My tab and public portfolios in a Public tab. Each row shows title, owner, block count, published state, creation date, and tags. The **+ New Portfolio** button opens the portfolio create page. ### Creating a portfolio The **Portfolio create** page (`{{BaseUrl}}/portfolios/new`) asks for title and optional description. After creation, the user is redirected to the portfolio detail page where blocks can be added. ### Portfolio detail page The **Portfolio detail** page (`/portfolios/{publicId}`) is the block editor. The page consists of: - A **Header** with title, owner, published state, creation and modification dates. - A **Block list** showing every block in display order. Each block is editable inline by the owner. - An **Add a block** toolbar below the block list, offering all block types in this order: - **Markup** — rich text with formatting. - **Image** — uploaded image with optional caption and alt text. - **Discovery** — a reference to a specific discovery, rendered as a card. - **Experiment** — a reference to a specific experiment, rendered as a card. - **Dataset** — a reference to a specific dataset, rendered as a card. - **Audio** — embedded audio player from Soundcloud, Spotify, or Apple Podcasts. - **Video** — embedded video player from YouTube. - **Analysis** — an AI-generated summary of all entities referenced in the portfolio. Costs tokens. - **Action buttons** in the header — Edit title, Publish/Unpublish, Share, Analyze, Delete. The block toolbar order is identical to the entity attach row in the Post and Message editors so that the three surfaces feel the same. ### Block types in detail #### Markup block A **Markup** block holds rich text with the following formatting: headings (H1-H4), bold, italic, strikethrough, ordered lists, unordered lists, blockquotes, code blocks, inline code, hyperlinks, and tables. Markup supports `@mentions` of other users. Mentioned users receive a notification. #### Image block An **Image** block holds an uploaded image. The image upload accepts PNG, JPEG, GIF, and WebP files. Each image block has optional **Caption** text shown below the image and optional **Alt Text** for screen readers. The image width on the page is controlled by the block's Width setting (Full or Half). #### Discovery / Experiment / Dataset block These reference blocks each hold a pointer to a specific entity owned by the portfolio author or made public by another user. The block renders the entity's title, description excerpt, key stats (correlation `r`, row count, etc.), and a thumbnail. Clicking the block navigates to the referenced entity's detail page. #### Audio block An **Audio** block embeds an audio player. The block accepts a URL from one of the following platforms: - Soundcloud (`soundcloud.com` or any `.soundcloud.com` subdomain) - Spotify (`open.spotify.com`) - Apple Podcasts (`podcasts.apple.com`) Each Audio block has: - A **URL** field where the source URL is pasted. - A **Show inline preview** checkbox. When checked, the player is embedded directly in the portfolio. When unchecked, the block renders as a small link card that opens the URL in a new tab. - A **Preview size** selector with three options: Small, Medium, Large. The selector is shown only when Show inline preview is checked. The size controls the player's vertical scale; horizontal width is controlled by the block's Width setting. #### Video block A **Video** block embeds a video player. The block accepts a URL from YouTube only (`youtube.com`, `www.youtube.com`, `youtu.be`, or `m.youtube.com`). A Video block has the same controls as an Audio block: URL field, Show inline preview checkbox, Preview size selector. YouTube embeds preserve a 16:9 aspect ratio; the preview size controls the maximum width (Small ~320 px, Medium ~560 px, Large ~800 px). #### Analysis block An **Analysis** block runs an AI analysis across every entity (datasets, experiments, discoveries) referenced elsewhere in the portfolio and produces a unified narrative summary. The block consumes tokens when generated. Re-running the analysis replaces the prior text. ### Block layout controls Every block has the following layout controls in its inline edit panel: - **Width** — Full (occupies the full page width) or Half (occupies half the page width, allowing two blocks side by side). - **Horizontal alignment** — Left, Center, Right. Affects how the block content aligns within its width. - **Vertical alignment** — Top, Middle, Bottom. Used when stacking half-width blocks of different heights. - **Font size** — for Markup blocks, an integer pixel size override. ### Drag-reordering blocks Blocks can be drag-reordered using the gripper handle on the left of each block. The new order is persisted immediately. ### Publishing a portfolio To publish a portfolio, open the portfolio's detail page (`/portfolios/{publicId}`) and click the **Publish** button in the header. You must be the portfolio's owner and signed in. Publishing changes the portfolio's published state from Private to Public. Step by step: 1. Go to the **Portfolios** page and open the portfolio you want to publish (or create one with **+ New Portfolio** and add at least one block first). 2. On the portfolio detail page, locate the **Publish** button in the header action buttons (next to Edit, Share, Analyze, and Delete). 3. Click **Publish**. The button hint reads "Publish to Home & Search". The portfolio's state badge changes to Public. Once published, the portfolio appears on the **Home** page feed and in **Search** results, and any signed-in or anonymous visitor can view it at its public URL. To make it private again, click the same button, which now reads **Unpublish** with the hint "Unpublish from Home & Search" — this removes it from the Home feed and Search and returns it to Private. Publishing visibility can also be set to a specific **Workgroup** instead of fully Public by using the **Share** button, which opens a workgroup picker. A workgroup-shared portfolio is visible only to members of that workgroup, not on the public Home feed. --- ## 10. Posts ### What a Post is A **Post** is a community discussion thread in the public feed. Every post has a subject (top-level posts only), a markdown body, an owner, optional image attachments, optional audio/video attachments, optional references to public datasets, experiments, discoveries, or portfolios, and optional tags. Posts can have replies, which are themselves posts with a parent. ### Posts list page The **Posts** page (`{{BaseUrl}}/posts`) shows the feed of top-level posts newest first. Each post card shows subject, body excerpt, owner, creation time, reply count, sentiment counts, tags, and attachment indicators. Clicking a card opens the post detail page. The page header has: - **+ New Post** button — opens the post editor. - **Search bar** — filters the feed by text in subject or body, or by tag (prefix the query with `#tagname`). - **Filter chips** — All / Mine / Unread. ### Post editor The **Post editor** dialog is opened by the **+ New Post** button on the Posts page, by the **Edit** button on the user's own post, or by the **Reply** button on any post. The editor has the following fields: - **Subject** — only shown on top-level posts. Optional but recommended. - **Message** — the post body, in markdown. Supports `@mentions` of other users. - **Images** — uploaded image attachments. Use the **Upload** link to attach one or more files, or paste an image directly into the editor (Ctrl/Cmd+V) to attach a screenshot. Pasted images upload only when Save is pressed, so a paste + cancel leaves no orphans. - **Attach** row — a horizontal row of buttons for attaching entities and media: - **Discoveries** — opens an entity picker listing public discoveries. - **Experiments** — opens an entity picker listing public experiments. - **Datasets** — opens an entity picker listing public datasets. - **Portfolios** — opens an entity picker listing public portfolios. - **Audio** — adds an empty Audio attachment slot to the editor. - **Video** — adds an empty Video attachment slot to the editor. - **Audio / Video attachments** — for each Audio or Video slot added, an inline editor with a URL field, a Show inline preview checkbox, and a Preview size selector. The URL must match the kind (Audio attachments only accept Soundcloud / Spotify / Apple Podcasts URLs; Video attachments only accept YouTube URLs). A live warning appears under the URL field if the host doesn't match. - **Tags** — type a tag and press Enter to add it. Click a tag chip to remove it. The **Save** button is disabled if no content has been entered (message, image, entity attachment, or media attachment), if any media attachment has an empty or kind-mismatched URL, or if a save is already in progress. ### Post detail page The **Post detail** page (`/posts/{publicId}`) shows the full post with all attachments rendered inline, followed by every reply nested under it. Each reply card shows the reply author, body, attachments, and its own reply button. Each post card has the following per-post controls visible to the appropriate roles: - **Reply** — open the post editor in reply mode. - **Edit** — opens the editor pre-filled (owner only). - **Delete** — permanently deletes the post or reply and everything nested under it. Available to the author; on replies, also to the **thread owner** (creator of the top-level post) and to Moderators, Curators, and Administrators. - **Hide / Unhide** (replies only) — the thread owner and Moderators+ can hide a reply without deleting it. A hidden reply stays in the thread as a *"[reply hidden by the thread owner or a moderator]"* placeholder so nested replies remain visible; the reply's author, the thread owner, and moderators still see the real content with an amber **Hidden** badge and can unhide it. Hidden replies do not appear in search. - **Pin to Profile** — pins the post to the top of the owner's public profile page (owner only, one post at a time). - **Featured** badge / toggle — marks the post as featured (Curator and Administrator roles only): an amber badge on the card plus inclusion in the Home page's **Featured** filter chip. For top-of-feed placement, Curators/Administrators **pin** the card instead (see Home Page). - **Sentiment** — thumbs up and thumbs down icons; the user's current choice is highlighted. - **Sentiment counts** — total positive and negative reactions visible to all viewers. ### Rendering of attachments on a post card Attachments on a post are rendered in this order: 1. The markdown message body, with auto-embedded media for any bare media URL on its own line. 2. Image carousel (left-arrow and right-arrow buttons to page through if more than one image). 3. Audio and Video attachments, each as either an embedded player (if Show inline preview is on) or a link card. 4. Entity attachment chips, in the order Discoveries, Experiments, Datasets, Portfolios. 5. Tags row. ### Pinning A user can pin one of their own posts to their public profile page using **Pin to Profile**. The pinned post appears at the top of the user's profile (`/users/{publicId}`) for every visitor. Each user can pin at most one post to their profile at a time; pinning a new post automatically unpins any previously-pinned one. The pin state updates immediately in the UI on click without waiting for the server round-trip. There is no user-level pin that targets the Home feed — only the administrator/curator **Featured** flag pushes posts to the top of the Home page. --- ## 11. Messages ### What a Message is A **Message** is a direct user-to-user communication. Unlike Posts, messages are not public. Each message has a sender, one or more recipients, a subject, a markdown body, optional image attachments, optional audio/video attachments, and optional references to public datasets, experiments, discoveries, or portfolios. Once sent, a message's content is immutable. ### Messages page The **Messages** page (`{{BaseUrl}}/messages`) shows the user's mailbox with three folder tabs: - **Inbox** — every message the user has received that has not been deleted. - **Sent** — every message the user has sent. - **Deleted** — every message the user has soft-deleted from Inbox or Sent. Each row in the message list shows sender or recipient names, subject, body excerpt, creation date, read/unread state, and attachment count. Clicking a row opens the message detail dialog. Unread messages are bolded. Above the list: - **+ New Message** button — opens the message composer. - **Filter chips** — All, Unread. - **Bulk action toolbar** — appears when one or more messages are selected via the row checkbox. Available bulk actions: Mark Read, Mark Unread, Move to Deleted, Restore (from Deleted folder), Delete Forever (from Deleted folder). ### Message composer The **Message composer** dialog is opened by the **+ New Message** button or by the **Reply** button on a received message. The composer has: - **To** — the recipient picker. Type a username to search; matching users appear in a dropdown; click to add. A message can have multiple recipients. - **Subject** — optional free text. - **Message** — the body, in markdown. Supports `@mentions`. - **Images** — uploaded image attachments. Use the **Upload** link to attach files or paste images directly (Ctrl/Cmd+V). - **Attach** row — same as in the Post editor. Includes Discoveries, Experiments, Datasets, Portfolios, Audio, and Video buttons. - **Audio / Video attachments** — same inline editor as in the Post editor. URL must match the kind, with the same Soundcloud / Spotify / Apple Podcasts / YouTube allowlist. The **Send** button is disabled until there is at least one recipient and at least one content field (subject, message, image, entity attachment, or media attachment). Audio/Video attachments with empty or kind-mismatched URLs also block sending. ### Message detail dialog The **Message detail** dialog shows the full message: sender, recipient list, send date, subject, body (markdown-rendered), every attachment in the same order as Posts (images, then audio/video, then entity chips). The dialog also has: - **Reply** button — opens the composer pre-filled with the original sender as a recipient and the original subject prefixed with "Re:". - **Move to Deleted** button — soft-deletes the message from the current folder (Inbox or Sent). Visible when the message is in Inbox or Sent. - **Delete Forever** button — permanently deletes the message. Visible only when viewing a message in the Deleted folder. Requires confirmation. Opening an unread Inbox message automatically marks it as read, which updates the notification bell count. --- ## 12. Workgroups ### What a Workgroup is A **Workgroup** is a named group of users used to share entities with a defined audience that is wider than just the owner but narrower than the public. Workgroups are the third visibility tier after Private and Public. ### Workgroups page The **Workgroups** page (`{{BaseUrl}}/workgroups`) lists every workgroup the user owns and every workgroup the user is a member of. Each row shows name, member count, owner, and creation date. The **+ New Workgroup** button opens a dialog that asks for name and optional description. The creator is automatically the owner and a member. ### Workgroup detail Clicking a workgroup row opens its detail view, which shows: - The workgroup name, description, owner, and creation date. - The **Members** list with each member's avatar, username, and role (Owner or Member). - The **Invite** button (owner only) — opens a user picker for adding members. - The **Remove** button next to each member (owner only) — removes that user. - The **Leave Workgroup** button (members only) — removes the current user from the workgroup. - The **Delete Workgroup** button (owner only) — permanently deletes the workgroup. Entities currently shared with this workgroup revert to the owner's Private state. ### Sharing with a workgroup The **Share** button on every Dataset, Experiment, Discovery, and Portfolio detail page opens a workgroup picker. Selecting a workgroup and confirming sets the entity's published state to Workgroup and assigns the chosen workgroup as the audience. Every member of the workgroup gains read access to the entity. --- ## 13. Profile ### Profile page The **Profile** page (`{{BaseUrl}}/profile`) is reached from the user menu in the header. The page is organized into four sections: #### Profile section Shows the user's avatar, username, first name, last name, role, member-since date, and bio. The owner can edit first name, last name, bio, and upload a new avatar image. #### Profile banner A wide header image shown at the top of the public profile page. Upload or replace it from the banner card (PNG / JPG / WebP, up to 25 MB; resized to 1500 px). After uploading, a **position slider** under the preview adjusts which part of the image shows inside the wide crop — drag toward "top" or "bottom" and the position saves automatically when you release. Replacing or removing the banner resets the position to centered. **Remove** deletes the image and falls back to the default gradient. #### Subscriptions section Shows the active subscription plan, monthly token allowance, renewal date, and payment method. Available actions: - **Upgrade** — change to a higher-tier plan, prorated for the remaining billing period. - **Downgrade** — change to a lower-tier plan, taking effect at the next billing cycle. - **Cancel** — cancel auto-renewal. The plan remains active until the end of the current billing period. - **Buy Tokens** — purchase a one-time token top-up at the listed rate. - **Payment History** — view past invoices. #### Security section - **Change Password** — opens a dialog asking for the current password and a new password. - **Verify Email** — re-sends the verification email if the account is unverified. - **Active Sessions** — lists every active sign-in session with device, IP address, and last activity timestamp. Each session has a **Sign out** button. #### Account section - **Delete Account** — opens a confirmation dialog. Confirming requests account deletion. All datasets, experiments, discoveries, portfolios, posts, and messages owned by the user are removed. ### Public profile Every user has a public profile page at `/users/{publicId}` showing their username, avatar, bio, member-since date, pinned post (if any), and recent public discoveries / portfolios / posts. Anonymous and Registered visitors can view this page. --- ## 14. Usage Dashboard The **Usage** page (`{{BaseUrl}}/usage`) is reached from the token balance in the header or from the user menu. The page shows: - **Current Token Balance** — available tokens at the top of the page. - **Monthly Usage** — a bar chart of token spend by day over the last 30 days. - **Per-Action Breakdown** — a table listing every billable action over the period, with timestamp, action type, target entity link, model used, input and output token counts, and total cost. - **Action Type Filter** — limit the breakdown to AI Search, Dataset Analysis, Experiment Analysis, Discovery Analysis, Portfolio Analysis, Ingestion, or all. - **Date Range Selector** — limit the breakdown to a custom date range. The page also shows a forecast of how long the current balance will last at the recent average spend rate. --- ## 15. Notifications ### Notification bell The **Notification bell** in the header shows the count of unread notifications as a small badge. Clicking the bell opens a panel listing the most recent notifications. ### Notification types Notifications are generated by the following events: - **@mention** — another user mentioned the recipient by `@username` in a post, post reply, or portfolio markup block. - **Comment on owned entity** — a user commented on one of the recipient's datasets, experiments, discoveries, or portfolios. - **Reply to comment** — someone replied to a comment the recipient had posted. - **New message** — the recipient received a new direct message. - **Workgroup invite** — the recipient was added to a workgroup. - **Ingestion complete** — a dataset or experiment owned by the recipient finished ingestion (success or failure). - **Subscription renewal** — the recipient's subscription successfully renewed or failed to renew. - **AI price change** — an AI model's pricing changed by more than the configured threshold (alerts the recipient about expected cost shifts). ### Notification panel actions - Clicking a notification opens its detail view; a **View …** link there navigates to the related entity (dataset, experiment, discovery, portfolio, post, or forum question). Rows that reference an entity also show a **jump arrow** (↗) for navigating directly without opening the detail view. Internal links inside a notification's text (for example the "Open:" link in a moderator notice) are clickable. - The **Mark all read** button at the top of the panel marks every notification as read in one action. - The **See all** link at the bottom opens the full notifications history page. --- ## 16. Search ### Header search bar The **Search bar** in the header searches public discoveries, experiments, datasets, portfolios, and posts by **title and description** (post body too). Only Public content is searchable for visitors who aren't the owner. - **As you type** (3+ characters), a dropdown shows the top quick-pick matches grouped by type — click one to jump straight to its detail page. The previous results stay on screen while a new query loads (you'll see a brief "Updating…"). - **Press Enter** to open the full results view on the Home page (`{{BaseUrl}}/?q=`), where matches render as the same rich feed panels the Home feed uses. See *Searching from the Home page*. ### How search matches and ranks Search is **term-based**, not exact-phrase: - Your query is split into individual words, and an item matches if it contains **any** of them. So "gold bananas" finds items mentioning gold *or* bananas — you don't lose results just because one word is missing. - Results are **ranked by relevance**: items containing **all** your words rank above items with only one, and an item where your words appear **in order as an exact phrase** ranks highest of all. Within the same relevance, each type keeps its natural order (discoveries by correlation strength, others by recency). - To require an **exact phrase**, wrap it in double quotes: `"gold prices"` matches only items containing that phrase, spaces and order preserved. You can mix quoted phrases with loose words. **Year search:** a bare four-digit year (e.g. `1970`) searches discoveries by the **time period their underlying data covers**, rather than matching the text "1970" in titles. --- ## 17. Browse by Topic The **Topics** item in the left sidebar opens a directory of the entire published corpus organized by subject. It is available to everyone, including signed-out visitors, and is a way to discover related public work by subject — complementary to Search. ### Topics directory Clicking **Topics** in the sidebar opens the directory, which lists every subject that published content has been filed under. Each subject shows how much content sits beneath it. ### Topic hub pages Clicking a subject opens its **hub page** (a URL like `{{BaseUrl}}/topics/{name}`). The hub page lists all the published discoveries, experiments, and datasets filed under that subject, paginated. Click any item to open its detail page. Topics are assigned automatically based on each item's content — you don't tag anything yourself. --- ## 18. Open Data and Exports Correlation Studio publishes its entire **public** corpus as open data that anyone can download and reuse. Only published content is ever included — private and unpublished content never appears in any export. ### Open Data page The **Open Data** page (`{{BaseUrl}}/data`) offers the whole public corpus as a single download, free to reuse under a **CC BY 4.0** license — you may reuse the data as long as you attribute the original author and Correlation Studio. The page offers two formats: - **Bulk NDJSON** — one record per public dataset, experiment, discovery, and portfolio, with its title, description, AI analysis, statistics, topic tags, author, and a link back to its page. Download it from the Open Data page or directly at `{{BaseUrl}}/open-data/correlation-studio-public.ndjson`. - **llms-full.txt** — the same corpus rendered as a single Markdown document, handy for feeding into AI tools. Available at `{{BaseUrl}}/llms-full.txt`. ### Exporting an individual dataset Any published dataset can be exported to **CSV without signing in** — use the **Export** button on the dataset's detail page. The exported file is also linked from the dataset as a downloadable distribution, so the data is easy to cite and reuse. --- ## 19. Tags A **Tag** is a free-text label attached to a Dataset, Experiment, Discovery, Portfolio, or Post. Tags are case-insensitive. Multiple entities can share the same tag. ### Adding and removing tags The tags row on every detail page shows current tags as chips. The owner can: - Type a new tag in the input next to the chips and press Enter to add it. If the tag does not exist anywhere in the system, it is created on the fly. - Click an existing tag chip on a detail page to remove it from that entity. ### Browsing tagged entities Clicking a tag chip on any entity card or detail page navigates to a filtered list showing every public entity carrying that tag. Tags also appear as filter options on the Home feed and Posts page. --- ## 20. Comments The comments section on every detail page (datasets, experiments, discoveries, portfolios, posts) lists every comment with author, time, body, and reply count. ### Posting a comment The comment input at the bottom of the comments section accepts free text or markdown. The **Post Comment** button submits. Submitting requires Registered role or above. Comments support `@mentions`. ### Replying to a comment Each comment has a **Reply** link. Clicking opens a reply input under the comment. Replies are nested one level deep. ### Editing and deleting comments - The author can edit their own comment via the **Edit** link. - The author can delete their own comment via the **Delete** link. - Moderators and Administrators can delete any comment. - Setting a sentiment (thumbs up / thumbs down) on a comment is done via the small reaction icons on the comment row. --- ## 21. Ratings The **Rating** widget on every Dataset, Experiment, Discovery, and Portfolio detail page lets the signed-in user assign a 1-to-5-star rating. The widget shows: - **Average rating** as the filled portion of five star outlines. - **Total rating count** in parentheses. - **Current user's rating** highlighted with a different color when the user has rated. Clicking a star sets the rating. Clicking the same star a second time clears the rating. --- ## 22. Sentiment The **Sentiment** widget on every detail page and on every Post / Post Reply / Comment lets the signed-in user set one of three sentiments: - **Positive** (thumbs up) — value +1. - **Neutral** — the absence of a positive or negative sentiment. Setting sentiment to Neutral is equivalent to clearing the user's prior choice. - **Negative** (thumbs down) — value -1. The widget shows the total count of Positive and Negative sentiments next to the icons. The user's current choice is highlighted. --- ## 23. @-Mentions Typing `@` in a post body, post reply body, message body, or portfolio markup block opens a picker showing matching usernames. Selecting a user inserts `@username` into the text. When the post / message / portfolio is saved, every newly-mentioned user receives an `@mention` notification. The mention text in the rendered markdown becomes a link to the mentioned user's public profile. Editing a post or message does not re-notify users who were already mentioned in the prior version. Only newly-added mentions notify. --- ## 24. AI Features Several features in Correlation Studio use AI models to generate text or search. Each AI feature consumes tokens from the user's balance. ### AI providers The application supports three AI search providers and one analysis provider: - **Claude** (Anthropic) — used for entity analyses (datasets, experiments, discoveries, portfolios) and as a search provider. - **Gemini** (Google) — used as a search provider in Remote Search mode. - **Grok** (xAI) — used as a search provider in Remote Search mode. ### AI dataset analysis The **Analyze** button on the dataset detail page sends the dataset's columns, summary statistics, and a sample of rows to Claude and asks for an interpretation. The result text is shown in the analysis section of the page. Costs tokens. ### AI experiment analysis The **Analyze** button on the experiment detail page sends the experiment's parameters, mean correlation, and top discoveries to Claude and asks for a unified interpretation. The result is shown in the analysis section. Costs tokens. ### AI discovery analysis The **Analyze** button on the discovery detail page sends the column pair, correlation coefficient, regression fit, and sample of paired points to Claude and asks for an interpretation. The result is shown in the analysis section. Costs tokens. ### AI portfolio analysis The **Analyze** button on the portfolio detail page sends every referenced entity's analysis (or summary if no analysis exists) to Claude and asks for a unified narrative tying them together. The result becomes an Analysis block in the portfolio. Costs tokens. ### Re-running and editing an analysis Once a **dataset** or **discovery** has an analysis, two things change in its AI Analysis panel: - The **Analyze** button becomes **Reanalyze** — running it again generates a fresh analysis that replaces the existing text (costs tokens, like the first run). - An **Edit** button appears beneath it. Edit opens the analysis text in an inline editor where you can refine or correct it by hand — fix a detail, add context the model missed, or tighten the wording — then **Save** (or **Cancel** to discard). Markdown is supported, and editing costs no tokens. When you save a hand-edited analysis, Correlation Studio adds a note to the entity's **Annotations** (for example, *"AI analysis manually edited by yourname on 2026-06-29 (UTC)."*) so the record plainly shows the text was human-adjusted. The model credit shown under the analysis is left unchanged, and your edited wording flows through to the entity's public page and to Corrie. Editing is available to the entity's owner — and, for discoveries, to Moderators and above. ### Remote Search The Dataset wizard's Remote Search mode queries Claude, Gemini, and Grok in parallel with the user's description, date range, and geography. Each provider returns candidate dataset URLs. The wizard displays the live provider status and the rejected URLs with reasons. Costs tokens. ### Token cost transparency Every AI action shows the estimated token cost before the user confirms. The actual cost is shown after the action completes and is logged in the Usage page. ### Refunds on failure If an AI action fails after tokens were charged (model error, network failure, etc.), the tokens are refunded automatically. The refund appears as a credit in the Usage page's per-action breakdown. --- ## 25. Drag and Drop Drag-reorder is available on the following lists: - **Datasets list page** — both My Datasets and Public Datasets tabs (Public tab reorder is local-only and not persisted server-side for non-owners). - **Experiments list page** — both the experiments and the discoveries shown as expanded sub-rows. - **Discoveries within an experiment** — on the experiment detail page list view. - **Columns on a dataset detail page** — reorders the columns' display index. - **Blocks in a portfolio** — reorders blocks via the gripper handle on each block. - **Reference order in a media attachment list** — reorders Audio and Video attachments in a post or message editor. In every case, the drag handle is the gripper icon on the left of the row. After a drag, the new order is sent to the server in a single batched call. --- ## 26. Multimedia Embeds Audio and Video can be embedded in three places: Portfolio blocks (Audio block / Video block), Post attachments (Audio / Video attach buttons), and Message attachments (Audio / Video attach buttons). All three surfaces accept the same set of platforms and behave the same way. ### Supported audio platforms - **Soundcloud** — host `soundcloud.com` or any subdomain `*.soundcloud.com`. Embed renders the Soundcloud player with the waveform visualization and play controls. - **Spotify** — host `open.spotify.com`. Supports tracks, albums, playlists, episodes, and shows. Embed renders the Spotify player at the matching height for the kind. - **Apple Podcasts** — host `podcasts.apple.com`. Any locale prefix is accepted (en-us, en-gb, ja-jp, etc.). Embed renders the Apple Podcasts player. ### Supported video platforms - **YouTube** — hosts `youtube.com`, `www.youtube.com`, `youtu.be`, and `m.youtube.com`. Embed renders the YouTube player at a 16:9 aspect ratio. ### Preview vs. link Every multimedia embed has a **Show inline preview** checkbox. When checked, the embed renders the platform's iframe player directly in the page. When unchecked, the embed renders as a small link card with a platform label and the URL; clicking the card opens the source in a new tab. ### Preview size When Show inline preview is checked, a **Preview size** selector chooses one of three sizes: - **Small** — minimum height. For Soundcloud ~166 px, Spotify ~80 px, YouTube ~320 px wide. - **Medium** (default) — moderate height. For Soundcloud ~305 px, Spotify ~152 px, YouTube ~560 px wide. - **Large** — maximum height. For Soundcloud ~450 px, Spotify ~352 px, YouTube ~800 px wide. ### Auto-embed in markdown Pasting a supported media URL on its own line in a Post body, Message body, or Portfolio markup block auto-embeds the player at the Medium size. This works without using the explicit Audio or Video attachment slots. Explicit attachments give more control over preview-vs-link and size; auto-embed is a convenience shortcut. --- ## 27. Sharing and Exporting ### Share button The **Share** button on every entity detail page opens a small dialog with two options: - **Copy link** — copies the entity's public URL to the clipboard. - **Share with Workgroup** — opens a workgroup picker. Selecting a workgroup and confirming sets the entity's published state to Workgroup with the chosen workgroup as the audience. ### Public links Every Public Dataset, Experiment, Discovery, and Portfolio has a stable URL at `/{type}/{publicId}` that can be shared with anyone. The link works for anonymous visitors. ### Embed and thumbnail URLs Published Discoveries have stable thumbnail image URLs that can be referenced from external sites. The thumbnail URL is shown in the discovery detail page's share menu. --- ## 28. Common Controls ### Search bar Every list page and the header have a search bar. The search bar is a single text input with a magnifying glass icon. The list updates as the user types (debounced). On the header search bar, pressing Enter or clicking the icon submits to the global search results page. ### Paginator Lists that exceed one page of results show a paginator at the bottom. The paginator has: - **Previous page** button. - **Page number buttons** — a window of nearby pages plus the first and last page. - **Next page** button. - **Total count** displayed next to the page controls. The current page is highlighted. ### Sortable table headers Tables with sortable columns show a chevron next to the active sort column. Clicking the column header sorts by that column; clicking it again reverses the direction. ### Drag handle The drag handle is the gripper icon (six small dots in a 2×3 grid) on the far left of a row or block. Hover changes the cursor to a move icon. Click and drag to reorder. ### Confirm dialogs Destructive actions (delete, unpublish, permanent delete) open a confirm dialog with: - **Title** describing the action. - **Message** describing what will happen. - **Severity** styling (warning or danger). - **Confirm** button (right) and **Cancel** button (left). The Escape key cancels the dialog. ### Visibility picker The Visibility picker is shown anywhere a user chooses among Private, Public, and Workgroup. The picker is a three-option radio group with a description under each option summarizing who can see the entity. ### Rating stars The rating stars widget is five star outlines. Hovering over a star fills it and the lower-numbered stars; clicking commits the rating. Hovering off without clicking restores the prior value. ### File drop zone The file drop zone is used in the Dataset wizard's Local Files step. It is a dashed-border rectangle that accepts drag-and-dropped files or, when clicked, opens the native file picker. Multiple files can be dropped at once. ### Status footer The status footer is a thin strip at the bottom of the page that shows transient activity (ingestion progress, save status, error messages). It is hidden when there is nothing to show. --- ## 29. Keyboard Shortcuts - **Escape** — close the topmost modal or confirm dialog. - **Enter** in a tag input — commit the typed tag. - **Ctrl/Cmd + Enter** in a message body or post body — submit the form. - **Ctrl/Cmd + V** in a Post or Message editor — paste an image from the clipboard as an attachment. - **/** — focus the header search bar (when not already in an input field). --- ## 30. Administration The **Administration** page (`{{BaseUrl}}/admin`) is reached from the user menu and is visible only to users with the Administrator role. The page has many tabs organized alphabetically; this section describes each tab. ### AI Dataset tab Lists every AI dataset search ever run, with timestamp, requesting user, providers used, total token cost, candidate URL count, and success status. Each row links to the original requesting user and to the resulting Dataset (if one was created). ### AI Discovery tab Lists every AI discovery analysis ever run, with timestamp, requesting user, model used, input and output token counts, total token cost, and the target discovery link. ### AI Models tab Manages the model lifecycle catalog. Each entry has a provider (Claude, Gemini, Grok), model name, status (Active, Deprecated, Retired), context window, and notes. Administrators add new models when providers release them, mark deprecating models for migration, and retire models that are no longer available. ### AI Portfolio tab Lists every AI portfolio analysis ever run, with timestamp, requesting user, portfolio link, model, token counts, and cost. ### AI Pricing tab Manages the per-model price for input tokens, output tokens, and grounding events (for search-capable models). Each cell shows the current price and the most recent change with timestamp. Updating a cell appends to the price history and dispatches user-facing notifications if the change exceeds the configured threshold percentage. ### AI Search tab Lists every AI dataset search across users with detailed per-provider breakdown. ### Analytics tab Product-growth reporting with durable history. Headline cards show registrations in the selected window, total users, weekly active users, stickiness (daily ÷ monthly active users), visitors on the last full day, and content published. Four charts — **Visitors** (unique human IPs, authenticated users, and bot IPs per day), **Registrations** (new accounts per day plus a cumulative total-users line), **Active users** (DAU / WAU / MAU), and **Corpus growth** (datasets, experiments, portfolios, and discoveries created or published per day) — over a selectable 30-day to all-time range. Below them, a **registration funnel** shows conversion from visitors through registration, first dataset, first experiment, publishing, and becoming a paying customer, for a selectable cohort window. Daily traffic aggregates are rolled up automatically every hour into permanent tables, so this history survives the raw request log's 30-day retention. Optional chart milestone markers (campaign launches, releases) come from the `AnalyticsMilestones` Configuration entry — a JSON array of `{"date": "2026-06-01", "label": "Public launch"}` objects. ### Appeals tab Lists appeal requests from users whose content was removed or whose accounts were restricted. Each appeal has a user, target entity, reason, timestamp, and status (Pending, Approved, Denied). Administrators can review and resolve appeals from this tab. ### Configuration tab Manages global system configuration values stored as name/value pairs. Examples include MaxConcurrentDatasetIngestions, AIPriceChangeNotifyThresholdPct, BaseUrl. Each row has an inline edit field and a Save button. ### Deletions tab Lists soft-deleted entities awaiting hard deletion by the cleanup sweep. Each row shows entity type, owner, deletion date, scheduled hard-delete date. Includes a **Recompute storage usage** action that reconciles user storage totals from the live entity tables. ### Email Blacklist tab Manages the list of email addresses blocked from registration. Each row has the address, reason, and date added. Administrators add and remove entries from this tab. ### Events tab A paginated log of significant system events (user registrations, subscription changes, AI failures, etc.) for audit and diagnostic purposes. ### Geocoding tab Shows statistics on the geocoding service that resolves geographic Text values into latitudes and longitudes. Includes cache hit rate, top requested locations, recent failures, and a per-user breakdown. ### Invoices tab Manages billing invoices generated by subscription renewals and token purchases. Each row has user, date, amount, status (Paid, Pending, Failed, Refunded), and download link. Administrators can issue refunds and download invoices in bulk. ### IP Blacklist tab Manages the list of IP addresses blocked from accessing the site. Each row has the address (or CIDR range), reason, and date added. ### Jobs tab Shows the live ingestion job queue: active jobs (running, queued), recent completions, and recent failures. Includes a **Pause Ingestion** toggle that stops new jobs from being dispatched (already-running jobs continue) and a **Pause Cleanup** toggle that pauses the dataset cleanup sweep. ### Ledger tab Shows the global token ledger: every credit, debit, refund, and adjustment with timestamp, user, action type, target entity, and balance after. ### Logs tab Shows connection logs from the web request middleware. Each row has timestamp, IP, user, HTTP method, request path, status code, duration, and the related server session. The page includes a filter input that searches the path column. Status codes are color-coded (red for 5xx, amber for 4xx). ### Monitoring tab Shows operational dashboards including AI provider margin analysis (revenue from user-paid actions minus actual upstream cost, grouped by provider over a date window). ### Privacy Policy tab Edits the Privacy Policy document shown on the public privacy page. Supports markdown editing and versioning. ### Promotions tab Manages promotional codes that grant token bonuses, subscription discounts, or free trials. Each code has a value, redemption limit, expiry, and usage count. ### Refunds tab Lists every refund issued for failed ingestions, failed AI actions, or administrative adjustments. Each row has the recipient, amount, reason, and timestamp. ### Runtime tab Provides live access to the application server's runtime file logs. The tab has two sub-tabs: - **Tail** — shows the current log file as it is written. Polls every 2 seconds for new bytes. The text area holds a rolling 5 MB window. Buttons: **Pause** stops polling (the text area freezes at the current position); **Resume** resumes polling and re-follows the file tail; **Refresh** forces an immediate poll; **Clear** empties the local text buffer. The status indicator next to the controls shows "live" or "paused", and counters show the current file size and buffer fill. - **History** — shows every prior daily log file. The left rail lists files grouped by month with the current file marked "live". Selecting a date loads its content into the right pane. Files larger than 10 MB load only the most recent 10 MB with a banner indicating truncation. Each loaded file has an inline **Download** link. The Runtime tab is visible only to Administrators. ### Server Session tab Shows the current server process's session information: process ID, start time, uptime, hostname, version, memory usage, and active connection count. ### Shards tab Shows the configured content shards (the application is shard-aware to scale entity storage). Each shard has a name, connection string label, storage used, entity counts, and health status. ### Stats tab Shows system-wide entity totals: public datasets, public experiments, public discoveries, public portfolios, public posts. Each row has a denormalized cached total and a live count. Includes a **Recompute system counters** action that reconciles the denormalized values against the live entity tables. ### Subscriptions tab Lists every active subscription with user, plan, renewal date, payment method, and lifetime revenue. Administrators can adjust plans, issue refunds, and cancel subscriptions from this tab. ### Terms of Service tab Edits the Terms of Service document shown on the public TOS page. Supports markdown editing and versioning. ### Usage tab Shows aggregate usage statistics: tokens consumed per day, per action type, per user. Includes filters for date range and action type. ### User Sessions tab Lists every active user authentication session. Each row has user, IP, user agent, sign-in time, and last activity. Includes a **Revoke** button per row to force sign-out of a specific session. ### Users tab Lists every user account. Each row has username, email, role, status (Active, Suspended, Deleted), subscription plan, token balance, storage used, registration date, and last activity. Available row actions: Promote / Demote role, Suspend / Reactivate, Reset Password, View User Details. ### Utilities tab Hosts one-shot recovery and reconciliation utilities. Each utility is idempotent and safe to re-run. Current utilities: - **Recompute storage usage** — walks every shard and writes the recomputed totals back to the central User.StorageUsedBytes field. - **Recompute system counters** — reconciles every denormalized counter from the live tables. - **Backfill failed-ingestion refunds** — issues refunds for any old failed datasets that were never refunded. - **Cleanup identity discoveries** — deletes every discovery with `r = 1.0` (analytically trivial) and any experiment left empty as a result, with the standard delete workflow (token reclamation, RAG removal, etc.). - **Discovery dedupe & experiment clean-up** — finds and removes a single user's effective-duplicate discoveries. Pick a user; the utility scans all their discoveries and groups any that share the exact same X and Y column (same orientation) — these are the duplicates that result from running the same cross-matrix experiment more than once. It presents each group as a checkbox list with the *newer* copies pre-checked for deletion (the **oldest is kept**, since it carries the established permalink, search/sitemap presence, thumbnail, published status, and any AI analysis). Deleting runs the regular delete workflow — token reclamation, storage free, RAG removal, thumbnail and home-feed cleanup, counters — and then deletes any parent experiment left with zero discoveries while recomputing the mean correlation on those that survive. (The per-experiment **Ignore duplicate discoveries** option prevents these duplicates from being created in the first place; this utility is for cleaning up history made before that option was used.) ### Workgroups tab Lists every workgroup in the system with name, owner, member count, and creation date. Administrators can rename, transfer ownership, and dissolve workgroups from this tab. --- ## 31. Statistics Glossary Plain-language definitions of the statistical terms you will encounter in Correlation Studio. Each entry explains what the term means, where it shows up in the UI, and how the product computes or uses it. Use this section when you are looking at a Discovery, a chart, or a Dataset quality readout and want to know what a particular number is telling you. Each entry is written to stand on its own — if Corrie surfaces just one definition in answer to a question, it should make sense without the surrounding section. ### 29.1 Correlation Measures **Pearson correlation coefficient (r).** Pearson r measures the strength and direction of a *linear* relationship between two numeric columns. It runs from −1 (perfect downward line) through 0 (no linear relationship) to +1 (perfect upward line). It is sensitive to outliers and to non-linear shapes — a tight U-curve can produce r ≈ 0 even though the two columns are clearly related. Correlation Studio computes Pearson r during experiment ingestion and shows it as the headline "r = …" on every Discovery card and Discovery detail page. The same value drives the green-to-amber-to-red color scale used across every chart type. See also: [Wikipedia — Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient). **Spearman rank correlation coefficient (ρ, rho).** Spearman ρ is Pearson's formula applied to the *ranks* of the values rather than the raw values. It uses the same −1 to +1 scale, but it captures any *monotone* relationship — a straight line, a curve, an S-shape — as long as Y consistently rises or falls as X rises. Because it works on ranks, it is much less sensitive to outliers than Pearson. Correlation Studio computes Spearman ρ alongside Pearson r on every Discovery and shows both on the Discovery detail page. When the two values disagree significantly, that gap is what powers the **divergence** indicator in the Bubble and Divergence charts. See also: [Wikipedia — Spearman's rank correlation coefficient](https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient). **p-value.** A p-value is the probability that you would see a correlation at least this strong by pure chance if the two columns were actually unrelated. Smaller is stronger: p = 0.01 means "about a 1 % chance this is noise"; p = 0.5 means "could easily be coincidence." Correlation Studio computes the p-value from a two-tailed Student-t test and shows it next to the correlation coefficient on the Discovery detail page, and in tooltips on the Bubble and Divergence charts. One thing to watch for: with large datasets (n in the thousands), almost any non-zero r produces a near-zero p-value. At that point the *size* of the effect matters more than the p-value. See also: [Wikipedia — p-value](https://en.wikipedia.org/wiki/P-value). **Confidence interval (95 %).** A confidence interval is a range around the reported correlation that the *true* correlation is likely to fall into, given the sample size you have. Correlation Studio reports it as `[lower, upper]` next to r on the Discovery detail page. A wide interval (for example `[0.1, 0.8]`) means "with this little data we cannot be sure"; a tight interval (for example `[0.71, 0.74]`) means "we are very confident the real r is around 0.72." The interval is computed using the Fisher z-transform and is not reported when the sample has 3 or fewer pairs. See also: [Wikipedia — Fisher transformation](https://en.wikipedia.org/wiki/Fisher_transformation). **Sample size (n).** Sample size n is the number of paired `(x, y)` data points that actually survived the join and the missing-value filter for a Discovery — *not* the row count of either source dataset on its own. A Pearson r of 0.95 with n = 4 is essentially a coincidence; the same r with n = 50,000 is genuine. The value is reported on every Discovery card and forms the Y-axis of the Bubble chart. Note: the chart drilldown is capped at 5,000 rows for rendering performance, but the n shown in the UI is always the full join count. **Divergence (Pearson vs Spearman).** Divergence is the gap `|ρ| − |r|` — how much better the rank-based Spearman score captures the relationship than the linear Pearson score. A large positive divergence flags a *non-linear monotone* relationship (a curve that always rises or always falls). A large negative divergence flags an *outlier-influenced* relationship (a few extreme points propping up Pearson without affecting the ranks). Correlation Studio uses a threshold of 0.15 to surface divergence as the small amber or blue dot next to discovery titles on the experiment list page, and as the entire X-axis of the Divergence chart. ### 29.2 Time-Series & Causality **Granger causality.** Granger causality is a statistical test of whether *past* values of one series help predict *future* values of another, beyond what the second series' own past already predicts. It is often described as "X Granger-causes Y," but it is *predictive* causality, not physical causation — it cannot tell you whether X actually drives Y or whether both are driven by some unobserved third factor. Correlation Studio reports two F-statistics, two p-values, and a direction (none, X→Y, Y→X, or bidirectional). It is run on demand from the **Run Granger Test** button on the Discovery detail page; it is not part of bulk experiment ingest because the cost is significant. Internally the service handles stationarity, picks an optimal lag via BIC, and compares restricted vs unrestricted regression fits. See also: [Wikipedia — Granger causality](https://en.wikipedia.org/wiki/Granger_causality). **Granger F-statistic.** The Granger F-statistic is the ratio of how much *additional* variance the lagged predictor explains over the baseline (own-history-only) model. A large F means the extra predictor really helped. Correlation Studio reports it separately for X→Y and Y→X on the Discovery's Granger panel. **Granger optimal lag.** The Granger optimal lag is the number of past time-steps that best predicted the present, chosen by minimizing the Bayesian Information Criterion across candidate lags. A lag of 1 means "yesterday's value is enough"; a lag of 7 on daily data means "the past week matters." It is stored on the Discovery and shown on the Granger panel. **Granger direction.** Granger direction is a one-byte summary of the bidirectional Granger result for a Discovery: none, X→Y, Y→X, or bidirectional. It drives the small arrow icon on the Discovery's Granger panel so you can tell at a glance which way the predictive influence runs. **Lag analysis.** Lag analysis is a non-Granger view of the same idea: it computes Pearson r at every shift of one series relative to the other, from −maxLag to +maxLag, and plots the result. The lag with the largest absolute r tells you the apparent lead-or-follow relationship between the two series. It is less rigorous than the Granger test but cheap and visual — it shows whether the relationship strengthens or weakens at any time offset. Available as the **Lag Analysis** chart view on the Discovery detail page. See also: [Wikipedia — Cross-correlation](https://en.wikipedia.org/wiki/Cross-correlation). **Rolling correlation.** Rolling correlation is Pearson r computed inside a sliding window of fixed size, swept across the joined series. It shows whether the relationship is *stable* over time or whether it strengthens, decays, or even flips direction. A flat band near 0.8 means "consistently strong"; a wild oscillation between +0.7 and −0.7 means the headline r is hiding a non-stationary relationship. Rendered as the **Rolling Correlation** chart view; the window size is user-adjustable. See also: [Investopedia — Correlation coefficient](https://www.investopedia.com/terms/c/correlationcoefficient.asp). **Autocorrelation.** Autocorrelation is the correlation of a series with its own lagged copy. Correlation Studio uses it internally as a sanity check on the Line Graph chart: if the lag-1 autocorrelation of both X and Y is below 0.3, the row order is essentially random noise and drawing the two series as overlaid lines would be misleading. In that case the chart shows an overlay warning suggesting Scatterplot or Trajectory mode instead. See also: [Wikipedia — Autocorrelation](https://en.wikipedia.org/wiki/Autocorrelation). **Stationarity.** A series is stationary if its statistical properties (mean, variance) do not change over time. Most time-series statistics — including Granger causality — assume stationarity. Correlation Studio's Granger service checks via a simplified Augmented Dickey-Fuller test; non-stationary series are automatically *differenced* (replaced with the values' first differences) up to twice before the test runs, so you do not need to pre-process the data yourself. ### 29.3 Regression **Regression (OLS).** Ordinary Least Squares regression fits a curve through a scatterplot that minimizes the sum of squared vertical distances from the curve to each point. Correlation Studio offers five models on the Discovery detail page: **Linear** (`y = a + bx`), **Polynomial degree 2 or 3**, **Logarithmic** (`y = a + b·ln(x)`), and **Exponential** (`y = a·e^(bx)`). Pick one from the regression model selector; the curve and its equation render as overlays on the scatter chart. See also: [Wikipedia — Ordinary least squares](https://en.wikipedia.org/wiki/Ordinary_least_squares). **R² (coefficient of determination).** R² is the fraction of variance in Y that the fitted regression curve explains, on a 0-to-1 scale. R² = 0 means the curve is no better than just guessing the mean; R² = 1 means the curve passes through every point exactly. For a linear fit, R² is simply r squared — but for polynomial, logarithmic, and exponential fits it tells you how well that *specific shape* matches the data. Correlation Studio shows R² alongside r in the Discovery header whenever a regression model is selected. See also: [Wikipedia — Coefficient of determination](https://en.wikipedia.org/wiki/Coefficient_of_determination). **Residual.** A residual is the signed distance from a data point to the fitted regression curve (`actual_y − predicted_y`). A positive residual means the point sits above the curve; negative means below. The **Residual Plot** visualization mode subtracts the fit from each Y value and plots the residuals against X — a healthy fit produces a structureless cloud around zero, while visible patterns (a curve, a fan-shape, a trend) mean the chosen regression model is missing something. See also: [Wikipedia — Errors and residuals](https://en.wikipedia.org/wiki/Errors_and_residuals). **95 % prediction interval.** A prediction interval is a band drawn above and below the regression curve indicating where *new* observations of Y at a given X are expected to land 95 % of the time. It is wider than a confidence interval on the mean because it accounts for both uncertainty in the fit *and* the spread of the data around the fit. Correlation Studio renders it as a light filled band in the same color as the discovery's r. See also: [Wikipedia — Prediction interval](https://en.wikipedia.org/wiki/Prediction_interval). ### 29.4 Distribution Statistics These appear on the **Distributions** tab and **Quality** tab of every Dataset detail page, one row per column. They are computed once during ingestion and read back from the precomputed column-stats table, so opening these tabs is fast even on large datasets. **Mean (average).** The mean is the sum of all values in a column divided by the count. It is the most familiar measure of center, but it is pulled hard by outliers — a single billion-dollar entry in an income column will drag the mean far above any individual real person's income. When a column is skewed, compare the mean to the median to see what is really going on. **Standard deviation (σ, std dev).** Standard deviation is a measure of spread — roughly, the "typical distance from the mean." A column with mean 100 and std dev 5 is tightly clustered; mean 100 and std dev 80 is wildly dispersed. Correlation Studio uses the sample standard deviation (denominator `n − 1`). See also: [Wikipedia — Standard deviation](https://en.wikipedia.org/wiki/Standard_deviation). **Minimum / Maximum.** The minimum and maximum are the smallest and largest non-null numeric values in a column. They define the X-axis bounds for the column histogram and the whiskers of the box plot. The same values are also surfaced on the Dataset card as **First Value** and **Last Value** for quick scanning. **Quartiles (Q1, median, Q3).** Quartiles split the sorted column into four equal-sized buckets. **Q1** is the value below which 25 % of the data sits; the **median** (Q2) is 50 %; **Q3** is 75 %. The median is robust to outliers in a way the mean is not — if you want a "typical" value from a skewed column, use the median. The gap between Q1 and Q3 is the IQR. See also: [Wikipedia — Quartile](https://en.wikipedia.org/wiki/Quartile). **IQR (interquartile range).** The IQR is `Q3 − Q1`, the spread of the middle 50 % of the data. It is used as the basis for outlier detection in Correlation Studio: any value below `Q1 − 1.5·IQR` or above `Q3 + 1.5·IQR` is flagged as a statistical outlier and contributes to the column's outlier count on the Quality tab. See also: [Wikipedia — Interquartile range](https://en.wikipedia.org/wiki/Interquartile_range). **Skewness.** Skewness is a measure of asymmetry. Skewness near 0 means the column is roughly symmetric (mean ≈ median); positive skew means a long tail to the right (a few very large values, like income or city population); negative skew means a long tail to the left. A high-skew column is often a hint that a log transform — available on the Tools page — will make the values easier to work with before correlation analysis. See also: [Wikipedia — Skewness](https://en.wikipedia.org/wiki/Skewness). **Histogram bins.** Histogram bins are a 20-bucket count of how many values fall in each equal-width slice of a column's range. Correlation Studio renders them as the bar chart on the Distributions tab. The shape of the distribution tells you a lot about what kind of column you are looking at: a bell shape is roughly normal, an exponential decay is power-law-ish, two humps mean the column is bimodal (two underlying populations mixed together). See also: [Wikipedia — Histogram](https://en.wikipedia.org/wiki/Histogram). **Outlier.** An outlier is a value far enough from the rest of the column to look suspicious. Correlation Studio uses the Tukey definition: anything outside `[Q1 − 1.5·IQR, Q3 + 1.5·IQR]`. The Quality tab shows the outlier count, the outlier percentage, and a sample of up to 10 of the most-distant values. Outliers are not removed automatically — they are surfaced as warnings. If you want to drop them, use the **Outlier Removal** transform on the Tools page. See also: [Wikipedia — Outlier](https://en.wikipedia.org/wiki/Outlier). ### 29.5 Aggregation & Join Concepts These terms appear in the Experiment creation wizard and on the Experiment detail page. They describe how the two datasets in an experiment are paired up before correlations are computed. **RowSequence join.** A RowSequence join pairs rows positionally: X row 1 with Y row 1, X row 2 with Y row 2, and so on. No key column is required. This is useful when the two datasets are already aligned — for example, side-by-side sensor readings sampled at the same instants — and you have no shared key to join on. It is the default for experiments where no obvious key match is detected. **SharedKey join.** A SharedKey join pairs rows by matching an exact value in a designated key column. Date and DateTime columns are stored as ISO 8601 strings so that `"2024-01-15"` on X joins `"2024-01-15"` on Y cleanly. When multiple X rows share the same key, they are aggregated using the column's chosen aggregate function before the join. Use SharedKey when both datasets share a categorical key (country code, product ID) or an exact-timestamp key. **TimeSeries join.** A TimeSeries join pairs rows whose key timestamps fall in the same time bucket. Both sides snap to the same epoch grid (hour, day, week, month, or year) controlled by a single tolerance knob (default 1 day). Use TimeSeries for joining time-series data sampled at different cadences — for example, a daily X dataset with hourly Y readings will roll up to daily buckets with the Y values averaged inside each day. Duplicates inside a bucket are aggregated using each column's chosen aggregate function. **Aggregate function (per column).** When a join collapses multiple rows into one bucket (a TimeSeries day, or SharedKey duplicates), each column's aggregate function decides how to combine them: **Avg** (default), **Sum**, **Min**, **Max**, **First**, **Last**, or **Count**. You set it per column in the Experiment creation wizard, so you can mix and match — Sum for sales, Avg for temperature, Last for stock price. **Sampling (percentage).** Sampling lets you run an experiment on a deterministic subset of rows. The sampler hashes each row's join key modulo 100 and keeps the rows below your chosen percentage. Both X and Y hash on identical key values, so the surviving pairs line up exactly — no broken pairs from independent random sampling. Re-running the same Experiment at the same percent produces the exact same sample, which is what makes published Discoveries stable across re-runs. **Correlation threshold (absolute).** The correlation threshold is the minimum absolute r value at which a column pair becomes a published Discovery. The default is 0.5, configurable per-experiment in the wizard. Pairs below the threshold are counted (you can see the skipped count on the experiment page) but do not produce a Discovery row or a drilldown dataset — this keeps each experiment focused on relationships strong enough to be worth showing. ### 29.6 Visualizations **Scatterplot.** The Scatterplot is the default Discovery view. It plots every joined `(x, y)` pair as a dot; the eye reads the correlation directly off the shape. The renderer is density-aware: dense clusters get larger dots so a thick blob does not collapse into a single black pixel. Best for exploring shape, spotting outliers, and getting an overall feel for the relationship. See also: [Wikipedia — Scatter plot](https://en.wikipedia.org/wiki/Scatter_plot). **Line graph.** The Line Graph view shows two overlaid lines (X on the left axis, Y on the right axis) plotted against row order. It is best for time-series data where row order is meaningful. The renderer is density-aware so it handles 30,000-segment renders without degenerating into solid bands. If the row order is essentially random noise (low autocorrelation in both X and Y), the chart will show an overlay warning suggesting you switch to Scatterplot or Trajectory mode. **Residual plot.** A Residual Plot is a scatter of `(x, residual)` after subtracting the fitted regression curve from each Y. It is used to *diagnose* a regression fit: a structureless cloud near zero means the model captured the relationship well; visible patterns mean it did not. Residuals are colored by magnitude — green near zero, amber further out, red far from zero. **Mini chart (sparkline).** A Mini Chart is a small, axis-free SVG line preview of a Discovery used in feed cards, grids, and the Experiments page's expandable rows. No axes, no labels — just the shape of the relationship at a glance. **Correlation heatmap.** The Correlation Heatmap is a 2-D grid of every column pair in an Experiment, with each cell colored by its r value (green to grey to red). It lets you see at a glance which clusters of variables move together. Click any cell to jump to that Discovery. See also: [Wikipedia — Heat map](https://en.wikipedia.org/wiki/Heat_map). **Bubble chart.** The Bubble Chart plots every Discovery in an Experiment with X = Pearson r, Y = sample size n, bubble area = the Pearson-vs-Spearman divergence, and color = correlation strength. Discoveries in the top-right corner are both *strong* and *well-sampled* — usually the most interesting ones. Large bubbles call out non-linear relationships worth a second look. **Divergence chart.** The Divergence Chart is a scatter of Pearson (x) vs Spearman (y) for every Discovery in an Experiment, with the y = x diagonal drawn as an "agreement line." Points on the diagonal mean the two metrics agree (a clean linear relationship). Points off the diagonal mean the relationship is non-linear or outlier-influenced, depending on which way it leans. **Correlation network.** The Correlation Network is a force-directed node-link diagram of every column appearing in an Experiment's Discoveries. Nodes are columns; edges are Discoveries; edge thickness and closeness reflect correlation strength. Use it to see column clusters and to spot "hub" columns that correlate with many others. **Trajectory plot.** A Trajectory Plot is a scatter mode that connects sequential points within the same group — for example, one country's progression through years — drawn as alpha-blended polylines. Use it for "phase space" views: how a `(GDP, life expectancy)` pair has evolved over time, with each country tracing its own arc. **Box plot.** A Box Plot is a compact summary of a column's distribution: a box from Q1 to Q3 (the IQR), a line at the median, whiskers out to the min and max, and dots for the outliers beyond `1.5 · IQR`. Shown side-by-side with the histogram on the Distributions tab of every Dataset. See also: [Wikipedia — Box plot](https://en.wikipedia.org/wiki/Box_plot). **Histogram (column).** A column Histogram is a bar chart of the 20-bucket value distribution for a single numeric column on the Distributions tab. The shape tells you what you are looking at — a bell is roughly normal, exponential decay suggests a power-law-ish distribution, two humps mean the column is bimodal. ### 29.7 Quality **Quality score (0–100).** The Quality Score is a single number summarizing dataset health, shown on the Data Quality tab. It starts at 100, then subtracts 15 per "error" warning (entirely-empty columns, more than 75 % missing values, type mismatches, large duplicate-row counts), 5 per "warning" warning (constant columns, moderate missingness, statistical outliers), and 1 per "info" warning, clamped to the range 0 to 100. It is a rule-of-thumb summary, not a rigorous statistical metric — the Quality tab also lists each underlying issue individually so you can decide what to address. **Duplicate rows.** Duplicate Rows is the count of rows whose entire value tuple matches another row. Correlation Studio counts duplicates during ingestion, surfaces the count on the Quality tab, and deducts from the quality score. Duplicates are not removed automatically — use the **Deduplicate** transform on the Tools page if you want to drop them. **Type mismatch count.** The Type Mismatch Count is the number of values in a column that did not parse as the column's declared datatype — for example, text in a numeric column, or unrecognizable garbage in a date column. These are detected during datatype inference and counted on the Quality tab so you can spot columns that need attention. **Cardinality (low / medium / high).** Cardinality is a categorical label for how many distinct values a column has, derived from the ratio `DistinctCount / TotalCount`. **Low** cardinality means a few categories (country, gender). **High** cardinality means nearly every row is unique (IDs, freeform text). The Quality tab uses this label to flag oddities — for example, "high-cardinality categorical" usually means a column has been misclassified and is really an identifier rather than a category. --- ## 32. Glossary of Product Terms - **Absolute Correlation Threshold** — the minimum `|r|` value for a column pair to produce a discovery in an experiment. - **Aggregate Function** — for Shared Key and Geo Type joins, the function used to aggregate numeric column values when multiple rows share a key value or location (Sum, Average, Min, Max, Count). - **Anonymous** — the role for users who are not signed in. - **Audio block** — a portfolio block that embeds an audio player from Soundcloud, Spotify, or Apple Podcasts. - **Block** — a single unit in a Portfolio. A portfolio is an ordered list of blocks. - **Catalog** — the topic-organized browser for public datasets. - **Column** — a named, typed field in a dataset. - **Comment** — a text message attached to a Dataset, Experiment, Discovery, Portfolio, or Post. - **Correlation Coefficient** — the Pearson `r` value computed for a pair of columns, between roughly -1 and +1. - **Curator** — the role allowed to manage the Dataset Catalog topic ontology. - **Datapoint** — the underlying paired X and Y values plus join keys for a single Discovery. - **Dataset** — a tabular dataset with columns and rows. - **Datatype** — the type of a column's values: Integer, Decimal, Date, DateTime, Text, Geo. - **Discovery** — one column-pair correlation result, owned by a parent Experiment. - **Experiment** — a pairing of two datasets that produces a set of Discoveries. - **Featured** — a Curator/Administrator editorial flag: an amber badge on the card + the Home page's Featured filter chip. Distinct from **Pinned** (below), which controls top-of-feed placement; user-level pinning targets the user's profile, not the Home page. - **File type** — the source file format: CSV, TSV, Excel, Google Sheets, HTML. - **Geo Data Type** — the geographic interpretation of a Text column: City, State, Country, Postal Code, Latitude, Longitude. - **Home Feed** — the layered feed of published content on the Home page (`{{BaseUrl}}/`): Pinned cards first, then repeating day-cycles of Trending bands → fresh community publishes → a daily-reshuffled random mix with experiment clusters, walking back through trending history until it becomes pure mix. - **Pinned (Home feed)** — a Curator/Administrator pin that holds a card at the very top of the Home feed, above Featured. Blue Pinned badge; set from the feed card's ⋯ menu or the pin button on entity detail pages. - **Trending band** — a full-width Home-feed section grouping public content that matches a topic currently trending in world news, produced by a daily AI job. Curators can hide bands or remove individual cards. - **Hint** — optional descriptive text for a column. - **Image block** — a portfolio block holding an uploaded image with caption and alt text. - **Ingestion** — the process of parsing a source file, populating rows, and indexing for correlation lookup. - **Join Key** — a column marked for use in a Shared Key or Time Series join. - **Join Tolerance** — the maximum allowed gap between paired timestamps in a Time Series join, in seconds. - **Join Type** — Row Sequence, Shared Key, Time Series, or Geo Type. - **Geo Type (join)** — pairs dataset rows on shared geography: the most specific GeoType column both datasets carry (Zipcode > City > FIPS > County > State > Province > Country > Continent) or a Latitude+Longitude pair. Rows sharing a location aggregate per column via the Aggregate Function. - **Mention** — `@username` syntax that notifies the named user. - **Message** — a private direct user-to-user communication. - **Moderator** — the role allowed to delete other users' comments and posts. - **Notification** — an in-app alert about an event affecting the recipient. - **Pin to Profile** — places one of the user's own posts at the top of their public profile page. The only user-level pin in the app; there is no user-level pin to the Home feed. - **Portfolio** — a block-based composition referencing datasets, experiments, discoveries, and embedding media. - **Post** — a community discussion thread in the public Posts feed. - **Preview Size** — the height bucket for an inline media player: Small, Medium, Large. - **Private** — a published state visible only to the owner. - **Public** — a published state visible to all users including anonymous visitors. - **Public ID** — the 10-character identifier used in URLs for users, datasets, experiments, discoveries, and portfolios. - **Published State** — Private, Public, or Workgroup. - **Rating** — a 1-to-5-star value set by a user on a Dataset, Experiment, Discovery, or Portfolio. - **Registered** — the role for signed-in standard users. - **Reply** — a Post or Message with a parent. - **Residual Plot** — a scatterplot of residuals after fitting a linear regression to a discovery's data. - **Row** — a single record in a dataset, holding one value per column. - **Row Sequence Join** — pairs rows by their position in the source file. - **Scatterplot** — the default visualization type; one dot per paired row. - **Sentiment** — Positive (+1), Neutral (0), or Negative (-1) reaction set by a user. - **Shared Key Join** — pairs rows where a designated key column has matching values. - **Source Type** — File, API, or SQL. - **Subscription** — a recurring billing plan that grants a monthly token allowance. - **Tag** — a free-text label attached to an entity. - **Time Series Join** — pairs rows by closest timestamp within the Join Tolerance. - **Thumbnail** — the small preview image for a Discovery shown on the Home feed and in sparkline views. - **Token** — the unit of cost for AI features. - **Topic** — a category in the Dataset Catalog ontology. - **Video block** — a portfolio block that embeds a YouTube video player. - **Visibility Picker** — the UI control for choosing Private, Public, or Workgroup. - **Visualization Type** — Scatterplot, Line Graph, or Residual Plot. - **Workgroup** — a named group of users with shared access to entities published with Workgroup visibility.