Python

How I Cut Millions of Useless Backtests by Filtering Combinations Before They Run

Three ranges. Nine values each. That’s 729 combinations. Sounds manageable, right? Now add two more indicator parameters. Suddenly you’re staring at 56 million combinations, your RAM is gone, and your backtest hasn’t even started.

I hit this wall while backtesting forex indicators. I was testing different parameter combinations for indicators like MACD — things like fast EMA period, slow EMA period, and signal line length. Each parameter had a range of possible values, and I needed to test every valid combination.

The problem? Most combinations were invalid. MACD requires signalLine < fastEMA < slowEMA. But Python’s itertools.product doesn’t care about that. It generates every combination, including the ones that make no mathematical sense. I was wasting time and memory on millions of combinations I’d immediately throw away.

Here’s how I fixed it: a Python generator that filters invalid combinations before they pile up in memory, and processes the valid ones in small batches. The result? Fewer combinations, less memory, and backtests that actually finish.

The Cartesian Product Problem

When you test indicator parameters, you define ranges for each one. For MACD, that might look like:

  • Fast EMA: 3 to 21
  • Slow EMA: 10 to 34
  • Signal Line: 1 to 16

To find the best settings, you need to test every valid combination. Python’s itertools.product gives you the cartesian product — every possible pairing across all ranges.

Here’s the basic approach using zip(*product(...)):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

import numpy as np
from itertools import product

range_1 = np.arange(1, 5, step=1, dtype=int)
range_2 = np.arange(1, 5, step=1, dtype=int)

combo_1, combo_2 = zip(*product(range_1, range_2))

print("combo_1:", combo_1)
print("combo_2:", combo_2)

The product() function produces the cartesian product — every possible pair from the two ranges. Then zip() regroups those pairs into separate tuples, one per parameter. The output:

1
2

combo_1: (1, 1, 1, 1, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 4)
combo_2: (1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4, 1, 2, 3, 4)

Each column is a unique combination. Four values in each range gives 16 combinations. That’s fine.

But what happens with real-world parameter ranges?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

import numpy as np

range_1 = np.arange(1, 50, step=1, dtype=int)    # 49 values
range_2 = np.arange(1, 50, step=1, dtype=int)    # 49 values
range_3 = np.arange(2, 30, step=1, dtype=int)    # 28 values
range_4 = np.arange(0.1, 30, step=0.1)           # 299 values
range_5 = np.arange(0.1, 30, step=0.1)           # 299 values

# This creates 5 lists, each with 56,538,748 elements
combo_1, combo_2, combo_3, combo_4, combo_5 = zip(*product(
    range_1, range_2, range_3, range_4, range_5
))

Over 56 million elements — per list. Your machine will choke on memory before the backtest even begins. And that’s regardless of which backtesting library you use. The bottleneck isn’t the library. It’s the sheer number of combinations.

Step 1: Process Combinations in Batches

Both product() and zip() are iterators. They generate values one at a time, on demand. The memory problem happens because zip(*product(...)) forces everything into memory at once.

The fix is a generator that collects combinations into small batches and yields them one group at a time:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

from itertools import product

def zip_product_in_batches(*iterables, size=1000):
    """
    Yields batches of zip(*product(...)) results,
    'size' combinations at a time.
    """
    if type(size) != int or size < 1:
        raise ValueError(
            f"'size' must be a positive integer. Got: {size}"
        )
    batch = []
    for elements in product(*iterables):
        batch.append(elements)
        if len(batch) >= size:
            yield zip(*batch)
            batch = []

    # Don't forget the last (possibly smaller) batch
    if batch:
        yield zip(*batch)

Instead of materializing all 56 million combinations, you process them 1,000 at a time (or whatever batch size fits your machine). Here it is in action:

1
2
3
4
5
6
7
8
9

import numpy as np

range_1 = np.arange(1, 5, step=1, dtype=int)
range_2 = np.arange(1, 5, step=1, dtype=int)

for combo_1, combo_2 in zip_product_in_batches(range_1, range_2, size=5):
    print("combo_1:", combo_1)
    print("combo_2:", combo_2)
    print("=" * 30)

Output:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

combo_1: (1, 1, 1, 1, 2)
combo_2: (1, 2, 3, 4, 1)
==============================
combo_1: (2, 2, 2, 3, 3)
combo_2: (2, 3, 4, 1, 2)
==============================
combo_1: (3, 3, 4, 4, 4)
combo_2: (3, 4, 1, 2, 3)
==============================
combo_1: (4,)
combo_2: (4,)
==============================

Same 16 combinations, but now they arrive in groups of 5. You process each batch, save the results, and move on. Memory stays flat no matter how many total combinations exist.

This alone makes it possible to run massive parameter sweeps on a regular laptop. But we can do much better.

Step 2: Filter Out Invalid Combinations

Batching solves the memory problem. But you’re still iterating over every single combination — including the invalid ones. For MACD, where signalLine < fastEMA < slowEMA must hold, most combinations from a broad parameter sweep are useless.

Take three ranges of 1–9 (like the signal line, fast EMA, and slow EMA). That’s 9 × 9 × 9 = 729 total combinations. But how many actually satisfy the ascending order constraint? Only 84. That means 88% of the work is wasted on combinations you’ll never use.

The solution: filter inside the generator, before combinations ever reach the batch.

There are two ways to do this, and which one you pick depends on your filtering rule.

When the Same Operator Applies to Every Pair

If every adjacent pair follows the same rule — like strict ascending order (el_1 < el_2 < el_3 < ...) — you can generalize it. Python’s built-in all() function checks whether every item in an iterable is true. That lets you apply one operator across any number of elements, so the function works with two or more parameter lists:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

from itertools import product

def filtered_zip_product_in_batches(*iterables, size=1000):
    """
    Yields batched zip(*product(...)) results, keeping only
    combinations where each element is strictly less than the next.
    Works with two or more iterables.
    """
    if type(size) != int or size < 1:
        raise ValueError(
            f"'size' must be a positive integer. Got: {size}"
        )
    batch = []
    num_iterables = len(iterables)
    for elements in product(*iterables):
        # Check: elements[0] < elements[1] < elements[2] < ...
        if all(elements[i] < elements[i+1] for i in range(num_iterables - 1)):
            batch.append(elements)
            if len(batch) >= size:
                yield zip(*batch)
                batch = []

    if batch:
        yield zip(*batch)

The key line is the all() check. It walks through the elements pairwise: is the first less than the second? Is the second less than the third? And so on. Only if every pair passes does the combination make it into the batch.

This works great when the same operator applies everywhere. But what if it doesn’t?

When Operators Differ Between Pairs

Sometimes the relationship between parameters isn’t the same across the board. Maybe the first must be strictly less than the second, but the second can be less than or equal to the third — like el_1 < el_2 <= el_3. You can’t express mixed operators in a single all() call, so spell out the parameters explicitly and write the condition by hand:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

from itertools import product

def filtered_zip_product_fixed(list_1, list_2, list_3, size=1000):
    """
    Batched zip-product for exactly 3 lists,
    keeping only combinations where list_1 < list_2 <= list_3.
    """
    batch = []
    for el_1, el_2, el_3 in product(list_1, list_2, list_3):
        # Skip invalid combinations immediately
        if not (el_1 < el_2 <= el_3):
            continue
        batch.append([el_1, el_2, el_3])
        if len(batch) >= size:
            yield zip(*batch)
            batch = []

    if batch:
        yield zip(*batch)

This gives you full control over each comparison. The downside is that the function is locked to a fixed number of parameter lists.

Seeing It in Action

Let’s test with three ranges of 1–9 and a batch size of 10:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

import numpy as np

range_1 = np.arange(1, 10, step=1, dtype=int)
range_2 = np.arange(1, 10, step=1, dtype=int)
range_3 = np.arange(1, 10, step=1, dtype=int)

for list_1, list_2, list_3 in filtered_zip_product_in_batches(
    range_1, range_2, range_3, size=10
):
    print("list_1:", list_1)
    print("list_2:", list_2)
    print("list_3:", list_3)
    print("=" * 25)

The first two batches look like this:

1
2
3
4
5
6
7
8

list_1: (1, 1, 1, 1, 1, 1, 1, 1, 1, 1)
list_2: (2, 2, 2, 2, 2, 2, 2, 3, 3, 3)
list_3: (3, 4, 5, 6, 7, 8, 9, 4, 5, 6)
=========================
list_1: (1, 1, 1, 1, 1, 1, 1, 1, 1, 1)
list_2: (3, 3, 3, 4, 4, 4, 4, 4, 5, 5)
list_3: (7, 8, 9, 5, 6, 7, 8, 9, 6, 7)
=========================

Every single combination satisfies list_1 < list_2 < list_3. No wasted rows.

The unfiltered zip_product_in_batches() would have yielded all 729 combinations. The filtered version yields just 84. That’s 88.5% fewer combinations — fewer iterations, less compute time, less memory.

For a MACD backtest with realistic parameter ranges, the savings are even more dramatic. Instead of burning hours on combinations where the fast EMA is larger than the slow EMA (which makes no sense), you skip them entirely.

Adapting the Filter for Your Use Case

The ascending-order filter (a < b < c) maps directly to MACD’s constraint where signalLine < fastEMA < slowEMA. But you can swap in any filtering logic.

For example, if you need a minimum gap between parameters:

1
2

# Keep only combinations where each value is at least 3 apart
if all(elements[i+1] - elements[i] >= 3 for i in range(num_iterables - 1)):

Or if you have a completely custom rule — say, the third parameter must be less than half the second:

1

if elements[2] < elements[1] / 2:

The pattern stays the same. Define your validity rule, check it before appending to the batch, and let the generator handle the rest.

Wrapping Up

Two small changes to how you generate parameter combinations can save you massive amounts of time and memory:

  1. Batch the output so you never hold all combinations in memory at once.
  2. Filter inside the generator so invalid combinations never reach your backtest.

For my forex indicator backtests, this turned an impossible parameter sweep into something I could run on a normal machine. The 729-to-84 reduction in the example above is modest — with real parameter ranges across multiple indicators, the invalid combinations easily outnumber the valid ones by 10:1 or more. How much you save will depend on your specific input ranges and filtering rules. Narrower ranges with strict constraints will cut more; wider ranges with loose rules will cut less.

The code is pure Python with no dependencies beyond itertools. Copy the filtered_zip_product_in_batches function, plug in your own filtering rule, and you’re set.

Disclaimer: This article is for educational and informational purposes only. It does not constitute financial, investment, or trading advice. The examples involving forex indicators are used purely to illustrate a programming technique. Any trading or investment decisions you make are solely your own responsibility. Always do your own research and consult a qualified financial advisor before making trading decisions.