It is a common problem when you have a worthy hypothesis (at least you believe it's worthy) about new features or improvements, but the production costs are sky-high, and there is no solid proof it would work. Moreover, your team is skeptical about it and expressing concerns (or vice versa). In that case, the final decision "to do or not to do” usually relies on someone's expertise and beliefs — but we all know how those are, in fact, unreliable.

So, the questions are: can we test if we should make something before we actually make it? and can we test cheap if the full realization costs a lot? I state: yes, we can — and much more often than it's usually considered.

Limited Tests

The simple part of this approach is that some features/improvements might be tested in a limited version; for example, if you think about rebalancing or narrative rewriting, you can change those just in the scope of the first 15-60min and just ignore the fact that it would lose consistency with a whole other game because the potential value of the experiment is much higher then any possible risk related to failed cohorts (and risk is probably exaggerated too).

It's not that much of a secret sauce, yet still, there is too much bias about avoiding risky experiments, even when the results might outweigh the expected risk a dozen times.

But sometimes, there is no way to make 5% of expensive features or improvements because their minimal viable realization is already expensive; then you try to:

Effects Tests

The idea is that you may test not the feature itself but separately test some of it's effects / factors / risks or pre-interests. General approach: decompose the initial feature on risk/value factors, prioritize them, and brainstorm on how they might be tested separately as part of other, much simpler features.

In the best case, you can find something that affects the same kind of experience and metrics, probably with lesser impact, but with results that might be interpreted and extrapolated to evaluate the initial feature's worth, at least to some extent.

At least you can find something to test one of the most crucial risks, pre-interest, or to receive any kind of result that might convince you if the initial feature is worth it or not.

Pre-interest

The simplest example: you have a game without guilds and ask yourself if this is a valuable feature. You can put “Guilds” button on the main screen and tell on tap that guilds require X level; also, there can be a mockup or list of guilds features (instead of simple "coming soon”). Next, you measure conversion from seeing this info to reaching X level versus those who didn't have the same button. The problem is that this kind of test tells you nothing about the value and impact of the Guild feature itself, but it's probably the only way to measure pre-interest (except cast-dev, but I can't advise relying on it). Such tricks shouldn't affect engagement much, so any result beyond statistical noise might be considered positive.

Proof of worth

Less relevant but still a simple example to understand the idea: it's always a good practice to use offer segmentation / adaptive offers for most games, especially those that depend on whales. Wanna proof without making an actual feature? Try a double offer.

YES, it's not a fully correct example, it's a different feature, targeting different goals, BUT it's also proof of the basic idea of segmentation: that there are users that are ready to spend either less or more for the basically same goods, offered in the same point of the game and differs by only a value — in case you had concerns, ofc 🤷🏻

Risk/Values Test

It's a rare case when you can actually test something on a much cheaper realization but still reliable results. I have one good example, but to provide that, I need more words and context to explain :)

Soooo Loooong Example

Once, I've worked with a classic time-manager. Core loop provides you with time-sensitive encounters/orders that should be solved by a chain of interactions with available objects, eg "the client wants a coffee and burger > you run to the coffee machine, start it, wait for coffee, get it, run to the kitchen for a pre-cooked burger, bring both to the client before his timer runs out and receive a reward, etc."

Time managers have level progression and light meta-game. Basically, all you can do at the meta layer is upgrade the objects you interact with: the coffee machine will make coffee faster, the kitchen will make burgers faster, or increase max burger capacity. Upgrade costs you the same soft currency earned on level progression. Objects for each level are pre-setted and always available. Monetization includes renewable attempts ('health'), on-level boosters and soft/hard used primary to upgrade objects in metagame.

Those of you with a game design background should already guess what is wrong here. The main problem is that when you're creating economic pressure by creating levels that require more currency for upgrades than you previously allow to earn — your paywall is too fucking obvious. Plus, core mechanics provide you with a lot less variability and randomness than match3's or similar. And, yeah, you can add some renewable sources of currency, but either at the expense of giving away your economy or by increasing the costs of new content production.

Initial Giga-Feature

I thought about a solution that would allow us to avoid content dependency by more effectively monetizing each separate piece of content and increasing liveops earn to cost ratio. Finally, I came up with those ideas:

1. To make on-level objects gacha-collectibles. So their upgrade would require not only currency but also X cards of this object, thus, we lessen the problem of too-transparent paywalls and might hide its real price. As a black-box gacha have the ability to shift players perception from "bastards! gave me coins for 7lvl coffee, but now require 8lvl!!11 😠" to "damn, need 1 more card for 8lvl coffee, instead got useless burgers cards…". Bonus advantage: possibility to create an asymmetric deficit when you do not simply lack X, but at the same time, you possess enough Y (lot of cards + lack of currency OR lack of cards + lot of curr.) — which more likely leads to payment even if its value/cost doesn't change.
2. We kept the same difficulty and win/lose ratio but rebuilt the algorithm of orders-generating to make it more variable, more random, but also adaptive. That would finally solve the problem of too obvious paywalls and open the way to the next point:
3. In-level objects are the most valuable content entities, so what if we make them not only upgradable with cards but also “unlockable”? — by default, new objects became available instant and free in the levels where it used for the first time. But this content is persistent, constant, visible, interactive, gameplay, and economy affecting; thus, it has the highest value for the player and might be the most desired piece of content either to purchase or achieve —so why don't we try to lock it by default? (so players will start the same levels with different objects available, which is possible only in combination with adaptive balance)

So, you get the idea. Both my (publishing) and studio product people agreed it has the potential to boost monetization. But all of us understood its costs (despite in the long-term it targeted cost reduction), and on top of that, the studio saw significant risk in ruining mid/late-game. Their main concerns were:

• core audience used to get objects for free and might get frustrated if we don't give it them;
• despite overall difficulty shouldn't change — players would get frustrated with failing more orders, some of which they utterly cannot solve cause they haven't unlocked the required object yet.

Fair concerns, I should say. Still, I believed we would earn much more than might lose. But how to prove that? Obviously, all that staff costs like hell.

Ultra Cheap Test

We discussed much, brainstormed, and at the end of the day, we mutually came up with the perfect idea: what if we test the main risks only? Not the whole feature with its gacha, adaptive balance, etc — but just the minimum that causes the same frustration, and let us know if that's a thing.

The idea was to put a new 'object' (Gum Machine) in all levels from 8 (~1-2d), lock it up on iap purchase (special offer), and add customers’ requirements for gum. Balance wasn't affected: you can ignore that object, fail all gum-orders, and win exactly as before. Buying GumMachine also gave minimal value and barely affected your winrate.

It's less than 10% prod-costs of the whole idea, but that simplification allows us to measure both the main risk that concerns the studio the most AND at least one of my hypotheses that visible-persistant objects should be sold well.

Now, try to guess how it affected the metrics of new users? No, I mean it — before opening a toggle, suggest some numbers, say it out loud, and only after that click below.

So, the hypothesis was proven, and the next step should be to implement the whole complex feature. And… it has never happened :) but that's a different story.