Custom data-processing node
1. Node overview
The custom data-processing node is a general-purpose Python data-processing module. As long as its input and output are pandas DataFrames, main() can perform any data-processing operation supported by Python, including machine-learning training, statistical modelling, label and signal generation, and multi-source fusion with pre-installed libraries such as NumPy, pandas, scikit-learn, LightGBM, and PyTorch. It accepts DataFrames from one or more upstream nodes and returns a new DataFrame for a downstream strategy node, AI model node, or custom trading logic node.
| Comparison | Data-processing node | Custom data-processing node |
|---|---|---|
| Configuration | Visual expressions and factor library | Python code editor |
| Expressiveness | Built-in factor functions and simple expressions | Full Python syntax plus pre-installed numpy, pandas, scipy, scikit-learn, lightgbm, optuna, torch, ta, TA-Lib, and other libraries |
| Use cases | Standard technical indicators, label assignment, and conditional filtering | Complex logic, custom factors, cross-period aggregation, machine-learning or statistical modelling, label and signal generation, and multi-source fusion |
| Learning curve | Low; no programming experience required | Moderate; requires basic pandas knowledge |
2. Capabilities and pre-installed libraries
2.1 Capability coverage
Any Python data processing is permitted as long as the input and output are DataFrames- Custom factor and indicator calculations: loops, state machines, dynamic windows, cross-symbol grouped iteration, and other logic that the DSL cannot express.
- Machine learning and statistical modelling: train or invoke
scikit-learn,scipy,lightgbm, ortorchmodels in the node and output predictions, probabilities, cluster labels, PCA components, and more. - Labels and supervised-learning targets: freely define future returns, class labels, ranking labels, triple-lag labels, and other complex targets.
- Multi-source fusion: receive several upstream branches through
datas[i]and perform outer merges, external joins, cross-period alignment, and cross-sectional ranking. - Signal composition and ensembles: combine predictions from several models or branches using voting, weighted averages, or time-axis smoothing.
- Data cleaning, anomaly detection, missing-value imputation, feature normalization, and dimensionality reduction, plus any other numpy or pandas operation.
ta_* functions, prefer the DSL to reduce maintenance. Use this node when the DSL cannot express one of the six capability groups above, particularly ML modelling, label generation, or signal ensembles.2.2 Pre-installed backend libraries
Import these libraries directly in main(); no installation is required| Category | Pre-installed libraries | Typical uses |
|---|---|---|
| Data processing | pandas, numpy, pyarrow, scipy, python-dateutil, pytz | DataFrame operations, vectorized computation, statistics, and time handling |
| Technical indicators | ta, TA-Lib | EMA, RSI, MACD, Bollinger Bands, and other indicators |
| Machine learning | scikit-learn, lightgbm, optuna | Classification, regression, clustering, dimensionality reduction, GBDT models, and hyperparameter search |
| Deep learning | torch (CPU; the GPU image also contains the CUDA build, matplotlib, and seaborn) | Custom neural networks and LSTM or Transformer training and inference |
| Other | requests, sqlalchemy, redis, pydantic, pyparsing | HTTP requests, database access, caching, and data validation |
statsmodels, xgboost, catboost, and tensorflow. Use scikit-learn, lightgbm, or torch instead.os, sys, subprocess, and socket, as well as calls such as eval, exec, and open. Non-compliant code cannot be saved.3. Interactive configuration
The panel below is the actual custom data-processing node configuration. Write processing logic directly in its code editor.
4. Code template
4.1 The main entry point
The required unified entry point, called by the system when the node runsfrom typing import List
import pandas as pd
def main(context, datas: List[pd.DataFrame]) -> pd.DataFrame:
...| Parameter / return value | Type | Description |
|---|---|---|
| context | UserContext | Context object providing services such as logging and file storage |
| datas | List[pd.DataFrame] | DataFrames returned by upstream nodes, in connection order |
| Return value | pd.DataFrame | Processed DataFrame containing at least timestamp and symbol |
4.2 Available context attributes
Context members available in a custom data-processing node| Attribute | Description |
|---|---|
| context.log | Logging service supporting info, warn, error, debug, detail, and dataframe |
| context.file | File service for loading or saving models and intermediate data |
| context.current_id | Current node ID as a string |
| context.current_config | Current node configuration dictionary populated from the frontend panel |
context.exchange or context.signal. To obtain trading pairs, read the input's symbol column, for example symbols = pd.concat(datas)['symbol'].unique().tolist().4.3 Input and output contract
How DataFrame columns correspond to node connection orderdatas[i] is the output of the node connected by the i-th incoming edge. The order is determined by the order in which connections were created on the canvas. With only one upstream node, use datas[0].| Column | Type | Required | Description |
|---|---|---|---|
| timestamp | int | Required | Millisecond timestamp |
| symbol | str | Required | Trading pair, such as BTCUSDT |
| prediction | float | Recommended | Default signal column read when passing data to a downstream Trader |
| Other columns | Any | Optional | Custom factor columns available to downstream AI models or strategy nodes |
5. Complete code examples
5.1 Example 1: Pass upstream data through
The smallest template for verifying node connections and data flowfrom typing import List
import pandas as pd
def main(context, datas: List[pd.DataFrame]) -> pd.DataFrame:
if not datas:
return pd.DataFrame()
df = datas[0]
context.log.info(f"Received {len(df)} upstream rows", source="Custom data processing")
return df5.2 Example 2: Dual-EMA crossover signal
Generate a prediction signal column (1 / 0 / -1) from bullish and bearish EMA crossoversfrom typing import List
import pandas as pd
def main(context, datas: List[pd.DataFrame]) -> pd.DataFrame:
df = datas[0].copy()
# Calculate EMAs separately by symbol so instruments are never mixed
df = df.sort_values(["symbol", "timestamp"])
grouped = df.groupby("symbol", group_keys=False)
df["ema_fast"] = grouped["close"].apply(lambda s: s.ewm(span=9, adjust=False).mean())
df["ema_slow"] = grouped["close"].apply(lambda s: s.ewm(span=21, adjust=False).mean())
# Bullish crossover=1 (long), bearish crossover=-1 (short), otherwise=0 (wait)
diff = df["ema_fast"] - df["ema_slow"]
prev = grouped["ema_fast"].shift(1) - grouped["ema_slow"].shift(1)
df["prediction"] = 0
df.loc[(diff > 0) & (prev <= 0), "prediction"] = 1
df.loc[(diff < 0) & (prev >= 0), "prediction"] = -1
context.log.info(
f"Signals generated: {int((df['prediction'] == 1).sum())} long, "
f"{int((df['prediction'] == -1).sum())} short",
source="Custom data processing",
)
return df[["timestamp", "symbol", "close", "ema_fast", "ema_slow", "prediction"]]5.3 Example 3: Merge multiple upstream inputs
Align and merge several upstream DataFrames on timestamp + symbolfrom typing import List
import pandas as pd
from functools import reduce
def main(context, datas: List[pd.DataFrame]) -> pd.DataFrame:
if not datas:
return pd.DataFrame()
# Outer-join every upstream DataFrame on timestamp + symbol
merged = reduce(
lambda left, right: pd.merge(left, right, on=["timestamp", "symbol"], how="outer"),
datas,
)
merged = merged.sort_values(["symbol", "timestamp"]).reset_index(drop=True)
# Simple weighted fusion: assume upstream frames have prediction columns and average them
pred_cols = [c for c in merged.columns if c.startswith("prediction")]
if pred_cols:
merged["prediction"] = merged[pred_cols].mean(axis=1)
context.log.info(
f"Merged data: {len(merged)} rows, columns={list(merged.columns)}",
source="Custom data processing",
)
return merged5.4 Example 4: Merge multiple models into a wide table
Common in multi-source strategies: each upstream branch is an independent model for a different instrument. Align their distinct prediction columns on timestamp and pass the resulting wide table downstream.[timestamp, symbol]. Its inputs are different feature sources for the same instruments, and each time/instrument pair remains one row while feature columns expand horizontally. This example performs a wide-table merge on [timestamp]. Its inputs are independent models for different instruments, producing one row per timestamp with columns such as pred_btc and pred_eth side by side. The downstream custom trading logic makes an independent decision for each instrument; see Example 3 in the custom trading logic node.from typing import List
import pandas as pd
def main(context, datas: List[pd.DataFrame]) -> pd.DataFrame:
"""Merge several model branches into one wide table.
datas[0] = BTC model output containing pred_btc
datas[1] = ETH model output containing pred_eth
Returns: a wide table with timestamp and each instrument's prediction column
"""
if not datas:
context.log.warn("datas is empty; returning an empty DataFrame", source="Custom data processing")
return pd.DataFrame()
df_btc = datas[0].copy() if len(datas) > 0 else pd.DataFrame()
df_eth = datas[1].copy() if len(datas) > 1 else pd.DataFrame()
# Graceful fallback: return the available branch so the pipeline can continue
if df_btc.empty:
return df_eth
if df_eth.empty:
return df_btc
# Keep only the ETH prediction and outer-join it to BTC, retaining every timestamp
eth_cols = [c for c in ["timestamp", "pred_eth"] if c in df_eth.columns]
merged = pd.merge(
df_btc,
df_eth[eth_cols],
on="timestamp",
how="outer",
suffixes=("", "_eth_dup"),
)
# Remove duplicate columns created when both branches contain the same names
dup_cols = [c for c in merged.columns if c.endswith("_eth_dup")]
if dup_cols:
merged.drop(columns=dup_cols, inplace=True)
merged.sort_values("timestamp", inplace=True)
merged.reset_index(drop=True, inplace=True)
context.log.info(
f"Wide-table merge complete: {len(merged)} rows, columns={list(merged.columns)}",
source="Custom data processing",
)
return mergedhow="outer" retains rows when timestamps differ; the missing instrument's prediction is NaN and downstream logic skips it with pd.notna(). ② suffixes plus removal of _eth_dup prevents redundant columns when the inputs share names. ③ Returning the other branch when one is empty lets the pipeline continue after a single-model failure.6. Output data structure
The custom data-processing node writes its DataFrame to the signal cache for downstream nodes:
- For a downstream AI model node, every column except
timestampandsymbolbecomes a model input feature. - A downstream strategy node can reference columns directly in entry and exit expressions.
- A downstream custom trading logic node reads the result from the
datasargument ofhandle_data(ctx, datas).
| timestamp | symbol | close | ema_fast | ema_slow | prediction |
|---|---|---|---|---|---|
| 2024-12-11 02:00:00 | BTCUSDT | 97800.0 | 97612.4 | 97480.2 | 1 |
| 2024-12-11 03:00:00 | BTCUSDT | 97950.0 | 97698.5 | 97515.6 | 0 |
| 2024-12-11 04:00:00 | BTCUSDT | 98150.0 | 97810.7 | 97572.4 | 0 |
7. FAQ
A: It follows connection order on the canvas: the first connected node is datas[0], the next is datas[1], and so on. Log datas[i].columns with context.log.info to verify the order.
A: No. It cannot access context.exchange or context.signal. Put trading logic in a strategy node or custom trading logic node.
A: Use df.groupby("symbol", group_keys=False) and calculate each instrument separately. Applying .shift() or .ewm() directly to the entire column serializes different instruments together and produces incorrect results.
A: Downstream AI, strategy, and Trader nodes rely on these columns as alignment keys. Without them, signals cannot be routed to the correct instrument and candlestick. Always return at least timestamp and symbol.
A: Call context.log.info or context.log.dataframe after important steps to inspect intermediate data. The backtest or paper-run log panel displays the complete stack trace. Do not use print(); use context.log.* consistently.