Workflow action

Data transform

ut_data

Select by JSON path, map, filter, sort, group and aggregate, pivot, de-duplicate and join lists of records, and convert CSV to JSON and back.

Officialv1.0.011 functions Pure computation

About

What it does

Data transform reshapes lists of records inside a workflow, the work that otherwise takes loops and a chain of collection nodes. Pick values out of an API response with a JSON path (order.lines[*].sku), rename and nest fields, filter with conditions such as status = paid and amount >= 500, sort on several keys, group by one or more fields with count, sum, average, minimum, maximum and distinct counts, pivot values into columns or unpivot columns into rows, and remove duplicates by chosen fields.

Join combines two lists on matching keys (inner, left, right or full), for example orders with the managers of their regions, and reports what did not match. CSV to JSON parses spreadsheet exports to RFC 4180 (quoted fields, embedded commas and line breaks, header row, number and boolean inference) and JSON to CSV writes lists back out with chosen columns, CRLF line endings and protection against formula injection. Everything runs in the sandbox with no network or data access.

Good for

  • Total and average order values per region before a weekly summary email
  • Join API results with a data table of owners and keep the unmatched rows for review
  • Import a CSV export, filter it and write a clean CSV back out

After installing

  1. Install the package: the action appears in Extensibility → Workflow actions and in the workflow designer's palette under Custom actions.
  2. Drag Data transform onto a workflow, choose the function and map Items to a list (a data table query result, an API response path or a JSON variable).
  3. Chain functions: map the {{prefix_items}} output of one step into the Items of the next, e.g. Filter, then Group & aggregate.
  4. Try the included "Sales by region (sample)" workflow from the API or its start form: it filters paid orders, groups them by region and writes the totals as CSV.

Examples

Real inputs, real outputs

Each example ran in the sandbox when this gallery was built; the outputs are exactly what the step returns.

Totals per region group

Inputs

{
  "items": [
    {
      "id": "SO-1",
      "region": "North",
      "product": "Desk",
      "status": "paid",
      "amount": 1200,
      "qty": 2
    },
    {
      "id": "SO-2",
      "region": "South",
      "product": "Chair",
      "status": "paid",
      "amount": 300,
      "qty": 3
    },
    {
      "id": "SO-3",
      "region": "North",
      "product": "Chair",
      "status": "open",
      "amount": 450,
      "qty": 5
    },
    {
      "id": "SO-4",
      "region": "North",
      "product": "Desk",
      "status": "paid",
      "amount": 600,
      "qty": 1
    },
    {
      "id": "SO-5",
      "region": "South",
      "product": "Lamp",
      "status": "void",
      "amount": 80,
      "qty": 2
    }
  ],
  "group_by": [
    "region"
  ],
  "aggregates": [
    "sum:amount",
    "avg:amount",
    "count_distinct:product"
  ]
}

Outputs

{
  "groups": [
    {
      "region": "North",
      "count": 3,
      "sum_amount": 2250,
      "avg_amount": 750,
      "count_distinct_product": 2
    },
    {
      "region": "South",
      "count": 2,
      "sum_amount": 380,
      "avg_amount": 190,
      "count_distinct_product": 2
    }
  ],
  "count": 2
}

Paid orders of 500 or more filter

Inputs

{
  "items": [
    {
      "id": "SO-1",
      "region": "North",
      "product": "Desk",
      "status": "paid",
      "amount": 1200,
      "qty": 2
    },
    {
      "id": "SO-2",
      "region": "South",
      "product": "Chair",
      "status": "paid",
      "amount": 300,
      "qty": 3
    },
    {
      "id": "SO-3",
      "region": "North",
      "product": "Chair",
      "status": "open",
      "amount": 450,
      "qty": 5
    },
    {
      "id": "SO-4",
      "region": "North",
      "product": "Desk",
      "status": "paid",
      "amount": 600,
      "qty": 1
    },
    {
      "id": "SO-5",
      "region": "South",
      "product": "Lamp",
      "status": "void",
      "amount": 80,
      "qty": 2
    }
  ],
  "conditions": [
    {
      "field": "status",
      "operator": "eq",
      "value": "paid"
    },
    {
      "field": "amount",
      "operator": "gte",
      "value": 500
    }
  ]
}

Outputs

{
  "items": [
    {
      "id": "SO-1",
      "region": "North",
      "product": "Desk",
      "status": "paid",
      "amount": 1200,
      "qty": 2
    },
    {
      "id": "SO-4",
      "region": "North",
      "product": "Desk",
      "status": "paid",
      "amount": 600,
      "qty": 1
    }
  ],
  "count": 2,
  "removed": 3
}

Products as columns pivot

Inputs

{
  "items": [
    {
      "id": "SO-1",
      "region": "North",
      "product": "Desk",
      "status": "paid",
      "amount": 1200,
      "qty": 2
    },
    {
      "id": "SO-2",
      "region": "South",
      "product": "Chair",
      "status": "paid",
      "amount": 300,
      "qty": 3
    },
    {
      "id": "SO-3",
      "region": "North",
      "product": "Chair",
      "status": "open",
      "amount": 450,
      "qty": 5
    },
    {
      "id": "SO-4",
      "region": "North",
      "product": "Desk",
      "status": "paid",
      "amount": 600,
      "qty": 1
    },
    {
      "id": "SO-5",
      "region": "South",
      "product": "Lamp",
      "status": "void",
      "amount": 80,
      "qty": 2
    }
  ],
  "rows": [
    "region"
  ],
  "column": "product",
  "value": "amount",
  "fill": 0
}

Outputs

{
  "items": [
    {
      "region": "North",
      "Desk": 1800,
      "Chair": 450,
      "Lamp": 0
    },
    {
      "region": "South",
      "Desk": 0,
      "Chair": 300,
      "Lamp": 80
    }
  ],
  "columns": [
    "Desk",
    "Chair",
    "Lamp"
  ],
  "count": 2
}

Orders with their region manager join

Inputs

{
  "left": [
    {
      "id": "SO-1",
      "region": "North",
      "product": "Desk",
      "status": "paid",
      "amount": 1200,
      "qty": 2
    },
    {
      "id": "SO-2",
      "region": "South",
      "product": "Chair",
      "status": "paid",
      "amount": 300,
      "qty": 3
    },
    {
      "id": "SO-3",
      "region": "North",
      "product": "Chair",
      "status": "open",
      "amount": 450,
      "qty": 5
    }
  ],
  "right": [
    {
      "region": "North",
      "manager": "Aroha"
    },
    {
      "region": "South",
      "manager": "Ben"
    }
  ],
  "left_key": [
    "region"
  ]
}

Outputs

{
  "items": [
    {
      "id": "SO-1",
      "region": "North",
      "product": "Desk",
      "status": "paid",
      "amount": 1200,
      "qty": 2,
      "manager": "Aroha"
    },
    {
      "id": "SO-2",
      "region": "South",
      "product": "Chair",
      "status": "paid",
      "amount": 300,
      "qty": 3,
      "manager": "Ben"
    },
    {
      "id": "SO-3",
      "region": "North",
      "product": "Chair",
      "status": "open",
      "amount": 450,
      "qty": 5,
      "manager": "Aroha"
    }
  ],
  "count": 3,
  "unmatched_left": 0,
  "unmatched_right": 0
}

Parse a CSV export csv_to_json

Inputs

{
  "csv": "id,name,amount\n1,\"Smith, Jo\",12.50\n2,Ana,7"
}

Outputs

{
  "items": [
    {
      "id": 1,
      "name": "Smith, Jo",
      "amount": 12.5
    },
    {
      "id": 2,
      "name": "Ana",
      "amount": 7
    }
  ],
  "columns": [
    "id",
    "name",
    "amount"
  ],
  "count": 2
}

Reference

Functions, inputs and outputs

The designer generates each function's input panel from these declarations; outputs appear in the data picker with their types.

Select (JSON path) select

Read a value from an object or list by path: a.b[0].c, items[-1] for the last item, items[*].name for every item's name (wildcards return a list).

Inputs

  • data json · required — The object or list to read from.
  • path string · required — For example order.lines[0].sku or order.lines[*].sku ($. prefix optional).

Outputs

  • value json — The value at the path (a list when the path has a wildcard; empty when not found).
  • values list of jsons — Every value found, as a list.
  • found boolean
  • count integer — How many values were found.

Map fields map

Pick, rename and nest fields of every item: "customer.name as customer", "total" or { "from": "a.b", "to": "c", "default": 0 }.

Inputs

  • items list of objects · required — A list of objects (rows), e.g. a data table query result or a JSON array.
  • fields list · required — One per field: source path, optionally "as" a new name (a dotted name creates nested objects), or objects { from, to, default }.
  • keep_other_fields boolean — Keep fields not listed (renamed ones move).

Outputs

  • items list of objects — The mapped items.
  • count integer — How many items were returned.

Filter filter

Keep the items that match conditions: eq, ne, gt, gte, lt, lte (numbers and dates compare as such), contains, not_contains, starts_with, ends_with, in, not_in, empty, not_empty; all must match or any.

Inputs

  • items list of objects · required — A list of objects (rows), e.g. a data table query result or a JSON array.
  • conditions list of objects · required — List of { "field": "status", "operator": "eq", "value": "paid" }; in / not_in take a list or comma-separated text.each: field (string), operator (string), value (json)
  • combine choice · all / any — Whether every condition or any condition must match.
  • case_sensitive boolean — Compare text exactly (off: ignore case).

Outputs

  • items list of objects — The matching items, in their original order.
  • count integer — How many items matched.
  • removed integer — How many items did not match.

Sort sort

Sort by one or more keys, ascending or descending, comparing as numbers, text or dates (auto: numbers when both values are numeric). Stable; empty values sort last.

Inputs

  • items list of objects · required — A list of objects (rows), e.g. a data table query result or a JSON array.
  • keys list · required — In priority order: "amount desc", "-amount", "name asc text", or { "field", "direction": "asc" | "desc", "type": "auto" | "number" | "text" | "date" }.

Outputs

  • items list of objects — The sorted items.
  • count integer — How many items were returned.

Group & aggregate group

Group items by one or more fields and compute aggregates per group: count, count_distinct, sum, avg, min, max, first, last or list. Each group row has the group fields, count and the aggregates.

Inputs

  • items list of objects · required — A list of objects (rows), e.g. a data table query result or a JSON array.
  • group_by list of strings · required — The field(s) to group by.
  • aggregates list — "sum:amount", "avg(amount) as average", "count_distinct:product" or { "op": "sum", "field": "amount", "as": "total" }. The default name is op_field (sum_amount).

Outputs

  • groups list of objects — One row per group, in order of first appearance: the group fields, count and each aggregate.
  • count integer

Pivot pivot

Turn values of one field into columns: one row per Rows key, one column per distinct Column value, aggregating Value (sum, count, avg, min, max, first, last).

Inputs

  • items list of objects · required — A list of objects (rows), e.g. a data table query result or a JSON array.
  • rows list of strings · required — Field(s) that identify an output row.
  • column string · required — Field whose values become columns.
  • value string — Field to aggregate in each cell (not needed for count).
  • aggregate choice · sum / count / avg / min / max / first / … — How to combine several values in one cell.
  • fill json — Value for a row/column with no items (default empty; e.g. 0).

Outputs

  • items list of objects — One row per Rows key with a property per column.
  • columns list of strings — The column values, in order of first appearance.
  • count integer — How many items were returned.

Unpivot unpivot

Turn columns into rows: for every item, one row per column with the column name and its value, keeping identifier fields.

Inputs

  • items list of objects · required — A list of objects with one column per measure.
  • keep_fields list of strings — Identifier fields copied to every row.
  • columns list of strings — The fields to turn into rows (default: every field not kept).
  • name_field string — Field that receives the column name.
  • value_field string — Field that receives the value.
  • skip_empty boolean — Leave out rows whose value is empty.

Outputs

  • items list of objects — One row per item and column.
  • count integer — How many items were returned.

Remove duplicates dedupe

Remove duplicate items, comparing chosen fields (or whole items, which also works for lists of plain values), keeping the first or the last of each.

Inputs

  • items list · required — A list of objects or plain values.
  • fields list of strings — Fields that make an item unique (default: the whole item).
  • keep choice · first / last — Which duplicate to keep (the order of the list is kept).
  • case_sensitive boolean — Off: "A@x.io" and "a@x.io" are duplicates.

Outputs

  • items list of jsons — The unique items.
  • count integer — How many items were returned.
  • removed integer — How many duplicates were removed.

Join join

Combine two lists on matching keys (inner, left, right or full outer). Right fields whose names clash with left fields get the prefix right_ (or your own prefix on every right field).

Inputs

  • left list of objects · required — The main list of objects.
  • right list of objects · required — The list to look up.
  • left_key list of strings · required — Key field(s) in the left list.
  • right_key list of strings — Key field(s) in the right list, in the same order (default: the left key names).
  • type choice · inner / left / right / full — Which unmatched rows to keep.
  • right_prefix string — Optional prefix for every field from the right list.
  • case_sensitive boolean — Off: keys match ignoring case.

Outputs

  • items list of objects — The joined rows (one per match).
  • count integer — How many items were returned.
  • unmatched_left integer — Left rows with no match.
  • unmatched_right integer — Right rows with no match.

CSV to JSON csv_to_json

Parse CSV text (RFC 4180: quoted fields, "" escapes, line breaks in quotes) into a list of objects, using the header row for field names and inferring numbers and true/false.

Inputs

  • csv text · required — The CSV text.
  • delimiter string — auto (detect), comma, semicolon, tab, pipe or the character(s).
  • header boolean — The first row holds the field names (off: column_1, column_2, …).
  • infer_types boolean — Turn numbers and true/false into numbers and booleans, empty cells into empty values (numbers with leading zeros stay text).
  • trim boolean — Trim spaces around values.

Outputs

  • items list of objects — One object per data row.
  • columns list of strings — The field names.
  • count integer — How many data rows.

JSON to CSV json_to_csv

Write a list of objects as CSV (RFC 4180 quoting, CRLF line endings by default), with chosen columns and headers. Text that a spreadsheet would run as a formula is escaped by default.

Inputs

  • items list of objects · required — A list of objects (rows), e.g. a data table query result or a JSON array.
  • columns list of strings — Fields to write, optionally "field as Header" (default: every field, in order of appearance).
  • delimiter string — Field separator (tab for tab-separated).
  • header boolean — Write the column names first.
  • line_ending choice · crlf / lf — CRLF (RFC 4180, Excel) or LF.
  • escape_formulas boolean — Prefix text starting with = + - @ with ' so spreadsheets do not run it (CSV injection).

Outputs

  • csv text
  • columns list of strings — The header names.
  • count integer — How many data rows.

Network

None. The action cannot make requests.

Data and files

None. It only sees its inputs.

Secrets

None.

Install

Install it in your workspace

Works on UserTasks cloud and any self-hosted instance. The action lands in Extensibility → Workflow actions, enabled, with its sample workflow.

In the app

  1. Open Extensibility → Workflow actions → Browse gallery and choose Install on Data transform. Or download workflow-action-data-transform-1.0.0.utpkg and drag it onto Automations → Packages.
  2. The wizard verifies the checksum, every file hash and the signature, and shows what it adds.
  3. In the workflow designer, drag Data transform from Custom actions, pick a function and fill its inputs. Save & Test shows the typed outputs.

With the CLI

shell
curl -fLO https://usertasks.com/gallery/workflow-action-data-transform-1.0.0.utpkg
usertasks packages install workflow-action-data-transform-1.0.0.utpkg
usertasks activities test ut_data -f '{"operation":"select"}'

Verify the download

File SHA-256
b53345e5b00f2bec410e92473940f049a177e501d56a898819cf39ea5baf55ed
Package checksum (what is signed)
sha256:fb15dff8f4a7c81dddb1e96479981369538feeb661be57f536eb03177b5cce1c
Module SHA-256
d0401d975e2f88d541f9ba45202facd27aaaf4e4f6775b0c41f14ae1c72b9119
Signer
UserTasks Gallery · 49:f3:04:90:47:02:4d:07:28:61:3d:dc:bb:aa:db:11

More workflow actions

All actions