Custom data-processing node

1. Node overview

The custom data-processing node is a general-purpose Python data-processing module. As long as its input and output are pandas DataFrames, main() can perform any data-processing operation supported by Python, including machine-learning training, statistical modelling, label and signal generation, and multi-source fusion with pre-installed libraries such as NumPy, pandas, scikit-learn, LightGBM, and PyTorch. It accepts DataFrames from one or more upstream nodes and returns a new DataFrame for a downstream strategy node, AI model node, or custom trading logic node.

Position in the data flow
Data sourceData processingCustom data processing[AI model]Custom trading logicTrading engine
How it differs from the data-processing node
ComparisonData-processing nodeCustom data-processing node
ConfigurationVisual expressions and factor libraryPython code editor
ExpressivenessBuilt-in factor functions and simple expressionsFull Python syntax plus pre-installed numpy, pandas, scipy, scikit-learn, lightgbm, optuna, torch, ta, TA-Lib, and other libraries
Use casesStandard technical indicators, label assignment, and conditional filteringComplex logic, custom factors, cross-period aggregation, machine-learning or statistical modelling, label and signal generation, and multi-source fusion
Learning curveLow; no programming experience requiredModerate; requires basic pandas knowledge
Recommended approach: configure standard indicators such as MA, RSI, and MACD quickly as expressions in the data-processing node. Put custom logic, such as multi-factor composition, cross-period alignment, and rule-based signals, in a custom data-processing node. The two node types can be connected in sequence.

2. Capabilities and pre-installed libraries

2.1 Capability coverage

Any Python data processing is permitted as long as the input and output are DataFrames
  • Custom factor and indicator calculations: loops, state machines, dynamic windows, cross-symbol grouped iteration, and other logic that the DSL cannot express.
  • Machine learning and statistical modelling: train or invoke scikit-learn, scipy, lightgbm, or torch models in the node and output predictions, probabilities, cluster labels, PCA components, and more.
  • Labels and supervised-learning targets: freely define future returns, class labels, ranking labels, triple-lag labels, and other complex targets.
  • Multi-source fusion: receive several upstream branches through datas[i] and perform outer merges, external joins, cross-period alignment, and cross-sectional ranking.
  • Signal composition and ensembles: combine predictions from several models or branches using voting, weighted averages, or time-axis smoothing.
  • Data cleaning, anomaly detection, missing-value imputation, feature normalization, and dimensionality reduction, plus any other numpy or pandas operation.
Selection guidance, not a hard constraint: if the requirement fits on one line in the data-processing node's factorExpression DSL using vectorized ta_* functions, prefer the DSL to reduce maintenance. Use this node when the DSL cannot express one of the six capability groups above, particularly ML modelling, label generation, or signal ensembles.

2.2 Pre-installed backend libraries

Import these libraries directly in main(); no installation is required
CategoryPre-installed librariesTypical uses
Data processingpandas, numpy, pyarrow, scipy, python-dateutil, pytzDataFrame operations, vectorized computation, statistics, and time handling
Technical indicatorsta, TA-LibEMA, RSI, MACD, Bollinger Bands, and other indicators
Machine learningscikit-learn, lightgbm, optunaClassification, regression, clustering, dimensionality reduction, GBDT models, and hyperparameter search
Deep learningtorch (CPU; the GPU image also contains the CUDA build, matplotlib, and seaborn)Custom neural networks and LSTM or Transformer training and inference
Otherrequests, sqlalchemy, redis, pydantic, pyparsingHTTP requests, database access, caching, and data validation
⚠ Not pre-installed: statsmodels, xgboost, catboost, and tensorflow. Use scikit-learn, lightgbm, or torch instead.
Security restrictions: static validation blocks unsafe imports such as os, sys, subprocess, and socket, as well as calls such as eval, exec, and open. Non-compliant code cannot be saved.

3. Interactive configuration

The panel below is the actual custom data-processing node configuration. Write processing logic directly in its code editor.

Custom data-processing node configuration

4. Code template

4.1 The main entry point

The required unified entry point, called by the system when the node runs
Function signature
python
from typing import List
import pandas as pd

def main(context, datas: List[pd.DataFrame]) -> pd.DataFrame:
    ...
Parameter / return valueTypeDescription
contextUserContextContext object providing services such as logging and file storage
datasList[pd.DataFrame]DataFrames returned by upstream nodes, in connection order
Return valuepd.DataFrameProcessed DataFrame containing at least timestamp and symbol

4.2 Available context attributes

Context members available in a custom data-processing node
AttributeDescription
context.logLogging service supporting info, warn, error, debug, detail, and dataframe
context.fileFile service for loading or saving models and intermediate data
context.current_idCurrent node ID as a string
context.current_configCurrent node configuration dictionary populated from the frontend panel
Important: custom data-processing nodes cannot access context.exchange or context.signal. To obtain trading pairs, read the input's symbol column, for example symbols = pd.concat(datas)['symbol'].unique().tolist().

4.3 Input and output contract

How DataFrame columns correspond to node connection order
Input: order of the datas list
datas[i] is the output of the node connected by the i-th incoming edge. The order is determined by the order in which connections were created on the canvas. With only one upstream node, use datas[0].
Output: returned DataFrame columns
ColumnTypeRequiredDescription
timestampintRequiredMillisecond timestamp
symbolstrRequiredTrading pair, such as BTCUSDT
predictionfloatRecommendedDefault signal column read when passing data to a downstream Trader
Other columnsAnyOptionalCustom factor columns available to downstream AI models or strategy nodes

5. Complete code examples

5.1 Example 1: Pass upstream data through

The smallest template for verifying node connections and data flow
python
from typing import List
import pandas as pd


def main(context, datas: List[pd.DataFrame]) -> pd.DataFrame:
    if not datas:
        return pd.DataFrame()
    df = datas[0]
    context.log.info(f"Received {len(df)} upstream rows", source="Custom data processing")
    return df

5.2 Example 2: Dual-EMA crossover signal

Generate a prediction signal column (1 / 0 / -1) from bullish and bearish EMA crossovers
python
from typing import List
import pandas as pd


def main(context, datas: List[pd.DataFrame]) -> pd.DataFrame:
    df = datas[0].copy()

    # Calculate EMAs separately by symbol so instruments are never mixed
    df = df.sort_values(["symbol", "timestamp"])
    grouped = df.groupby("symbol", group_keys=False)
    df["ema_fast"] = grouped["close"].apply(lambda s: s.ewm(span=9, adjust=False).mean())
    df["ema_slow"] = grouped["close"].apply(lambda s: s.ewm(span=21, adjust=False).mean())

    # Bullish crossover=1 (long), bearish crossover=-1 (short), otherwise=0 (wait)
    diff = df["ema_fast"] - df["ema_slow"]
    prev = grouped["ema_fast"].shift(1) - grouped["ema_slow"].shift(1)
    df["prediction"] = 0
    df.loc[(diff > 0) & (prev <= 0), "prediction"] = 1
    df.loc[(diff < 0) & (prev >= 0), "prediction"] = -1

    context.log.info(
        f"Signals generated: {int((df['prediction'] == 1).sum())} long, "
        f"{int((df['prediction'] == -1).sum())} short",
        source="Custom data processing",
    )
    return df[["timestamp", "symbol", "close", "ema_fast", "ema_slow", "prediction"]]

5.3 Example 3: Merge multiple upstream inputs

Align and merge several upstream DataFrames on timestamp + symbol
python
from typing import List
import pandas as pd
from functools import reduce


def main(context, datas: List[pd.DataFrame]) -> pd.DataFrame:
    if not datas:
        return pd.DataFrame()

    # Outer-join every upstream DataFrame on timestamp + symbol
    merged = reduce(
        lambda left, right: pd.merge(left, right, on=["timestamp", "symbol"], how="outer"),
        datas,
    )
    merged = merged.sort_values(["symbol", "timestamp"]).reset_index(drop=True)

    # Simple weighted fusion: assume upstream frames have prediction columns and average them
    pred_cols = [c for c in merged.columns if c.startswith("prediction")]
    if pred_cols:
        merged["prediction"] = merged[pred_cols].mean(axis=1)

    context.log.info(
        f"Merged data: {len(merged)} rows, columns={list(merged.columns)}",
        source="Custom data processing",
    )
    return merged

5.4 Example 4: Merge multiple models into a wide table

Common in multi-source strategies: each upstream branch is an independent model for a different instrument. Align their distinct prediction columns on timestamp and pass the resulting wide table downstream.
How this differs from Example 3: Example 3 performs a long-table merge on [timestamp, symbol]. Its inputs are different feature sources for the same instruments, and each time/instrument pair remains one row while feature columns expand horizontally. This example performs a wide-table merge on [timestamp]. Its inputs are independent models for different instruments, producing one row per timestamp with columns such as pred_btc and pred_eth side by side. The downstream custom trading logic makes an independent decision for each instrument; see Example 3 in the custom trading logic node.
python
from typing import List
import pandas as pd


def main(context, datas: List[pd.DataFrame]) -> pd.DataFrame:
    """Merge several model branches into one wide table.
    datas[0] = BTC model output containing pred_btc
    datas[1] = ETH model output containing pred_eth
    Returns: a wide table with timestamp and each instrument's prediction column
    """
    if not datas:
        context.log.warn("datas is empty; returning an empty DataFrame", source="Custom data processing")
        return pd.DataFrame()

    df_btc = datas[0].copy() if len(datas) > 0 else pd.DataFrame()
    df_eth = datas[1].copy() if len(datas) > 1 else pd.DataFrame()

    # Graceful fallback: return the available branch so the pipeline can continue
    if df_btc.empty:
        return df_eth
    if df_eth.empty:
        return df_btc

    # Keep only the ETH prediction and outer-join it to BTC, retaining every timestamp
    eth_cols = [c for c in ["timestamp", "pred_eth"] if c in df_eth.columns]
    merged = pd.merge(
        df_btc,
        df_eth[eth_cols],
        on="timestamp",
        how="outer",
        suffixes=("", "_eth_dup"),
    )

    # Remove duplicate columns created when both branches contain the same names
    dup_cols = [c for c in merged.columns if c.endswith("_eth_dup")]
    if dup_cols:
        merged.drop(columns=dup_cols, inplace=True)

    merged.sort_values("timestamp", inplace=True)
    merged.reset_index(drop=True, inplace=True)

    context.log.info(
        f"Wide-table merge complete: {len(merged)} rows, columns={list(merged.columns)}",
        source="Custom data processing",
    )
    return merged
Key points:how="outer" retains rows when timestamps differ; the missing instrument's prediction is NaN and downstream logic skips it with pd.notna(). ② suffixes plus removal of _eth_dup prevents redundant columns when the inputs share names. ③ Returning the other branch when one is empty lets the pipeline continue after a single-model failure.

6. Output data structure

The custom data-processing node writes its DataFrame to the signal cache for downstream nodes:

  • For a downstream AI model node, every column except timestamp and symbol becomes a model input feature.
  • A downstream strategy node can reference columns directly in entry and exit expressions.
  • A downstream custom trading logic node reads the result from the datas argument of handle_data(ctx, datas).
timestampsymbolcloseema_fastema_slowprediction
2024-12-11 02:00:00BTCUSDT97800.097612.497480.21
2024-12-11 03:00:00BTCUSDT97950.097698.597515.60
2024-12-11 04:00:00BTCUSDT98150.097810.797572.40

7. FAQ

Q: How is the order of datas determined when there are several upstream nodes?

A: It follows connection order on the canvas: the first connected node is datas[0], the next is datas[1], and so on. Log datas[i].columns with context.log.info to verify the order.

Q: Can this node access exchange market data or place orders?

A: No. It cannot access context.exchange or context.signal. Put trading logic in a strategy node or custom trading logic node.

Q: How should I calculate by symbol when one DataFrame contains several instruments?

A: Use df.groupby("symbol", group_keys=False) and calculate each instrument separately. Applying .shift() or .ewm() directly to the entire column serializes different instruments together and produces incorrect results.

Q: What happens if the output lacks timestamp or symbol?

A: Downstream AI, strategy, and Trader nodes rely on these columns as alignment keys. Without them, signals cannot be routed to the correct instrument and candlestick. Always return at least timestamp and symbol.

Q: How do I troubleshoot an execution error?

A: Call context.log.info or context.log.dataframe after important steps to inspect intermediate data. The backtest or paper-run log panel displays the complete stack trace. Do not use print(); use context.log.* consistently.

    BeeQuant - AI Quantitative Trading Platform