tl;dr

• Box storage is a new type of storage for apps. An app can create boxes on-demand, as many boxes as it needs.
• Boxes can only be manipulated and read on-chain by the app that created them.
• When making an app call that uses boxes, the boxes must be referenced in the new boxes array, which functions analogously to the foreign apps/accounts/assets arrays. These references give access to the boxes across the transaction group.
• Creating boxes entails an increase to the app account’s minimum balance requirement.
• A series of new TEAL opcodes have been added to interact with boxes from a Smart Contract: box_create, box_put, box_replace, box_get, box_extract, box_len, box_del.

Introduction

Storage on blockchain is expensive, and for good reason: every byte of data is stored on thousands of machines - every computer participating in the blockchain. The model for storage is therefore a carefully designed system, fulfilling the needs of blockchain applications while guaranteeing performance.

On the Algorand blockchain, there are several different types of storage. There’s storage for tracking Algo balances, for tracking ASA balances, for account information, and for Smart Contracts.

Today, we introduce a new type of storage for Algorand Smart Contracts: boxes. A Smart Contract can have an essentially unlimited amount of boxes, which is a new and powerful paradigm for Algorand.

This article covers the technical aspects of this upgrade.

Local, Global, and Box Storage

Smart Contracts (A.K.A. “apps”) can now have three different types of storage: local storage, global storage, and box storage.

Global state and boxes are associated with the app itself, whereas local state is associated with each user account that opts into the application. Each storage option’s properties are described below.

Global state for app a:

• Allocation:
  • Can include between 0 and 64 key/value pairs and a total of 8K of memory to share among them.
  • The amount of global storage is allocated in k/v units, and determined at contract creation. This cannot be edited later.
  • The contract creator address is responsible for funding the global storage (by an increase to their minimum balance requirement, see below).
• Reading:
  • Can be read by any app call that has specified app a’s ID in its foreign apps array.
  • Can be read on-chain using the k/v pairs defined (from off-chain, can be read using goal or APIs + SDKs).
• Writing:
  • Can only be written by app a.
• Deletion:
  • Is deleted when app a is deleted. Cannot otherwise be deallocated (though of course the contents can be cleared by app a, but this does not change the minimum balance requirement).

Local state for account x for app a:

• Allocation:
  • Is allocated when account x opts in to app a (submits a transaction to opt-in to app a).
  • Can include between 0 and 16 key/value pairs and a total of 2KB of memory to share among them.
  • The amount of local storage is allocated in k/v units, and determined at contract creation. This cannot be edited later.
  • The opted-in user address is responsible for funding the local storage (by an increase to their minimum balance).
• Reading:
  • Can be read by any app call that has app x in its foreign apps array and account x in its foreign accounts array.
  • Can be read on-chain using the k/v pairs defined (from off-chain, can be read using goal and the SDKs).
• Writing:
  • Is editable only by app a, but is delete-able by app a or the user x (using a ClearState call, see below).
• Deletion:
  • Deleting an app does not affect its local storage.
  • Clear state. Every Smart Contract on Algorand has two programs: the approval and the clear state program. An account holder can clear their local state for an app at any time (deleting their data and freeing up their locked minimum balance). The purpose of the clear state program is to allow the app to handle the clearing of that local state gracefully.
  • Account x can request to clear its local state using a close out transaction.
  • Account x can clear its local state for app a using a clear state transaction, which will always succeed, even after app a is deleted.

Boxes for app a:

• Allocation:
  • App a can allocate as many boxes as it needs, when it needs them.
  • App a allocates a box using the box_create opcode in its TEAL program, specifying the name and the size of the box being allocated.
    • Boxes can be any size from 0 to 32K bytes.
    • Box names must be at least 1 byte, at most 64 bytes, and must be unique within app a.
  • The app account is responsible for funding the box storage (with an increase to its minimum balance requirement, see below for details).
  • A box name must be referenced in the boxes array of the app call to be allocated.
• Reading:
  • App a is the only app that can read the contents of its boxes on-chain. This on-chain privacy is unique to box storage. Recall that everything can be read by anybody from off-chain using the algod or indexer APIs.
  • To read box b from app a, the app call must include b in its boxes array.
  • Read budget: Each box reference in the boxes array allows an app call to access 1K bytes of box state - 1K of “box read budget”. To read a box larger than 1K, multiple box references must be put in the boxes arrays.
    • The box read budget is shared across the transaction group.
    • The total box read budget must be larger than the sum of the sizes of all the individual boxes referenced (it is not possible to use this read budget for a part of a box - the whole box is read in).
  • Box data is unstructured. This is unique to box storage.
  • A box is referenced by including its app ID and box name.
• Writing:
  • App a is the only app that can write the contents of its boxes.
  • Exactly analogous to reading, each box ref in the boxes array allows an app call to write 1K bytes of box state - 1K of “box write budget”.
• Deletion:
  • App a is the only app that can delete its boxes.
  • If an app is deleted, its boxes are not deleted. (the correct cleanup design is to look up the boxes from off-chain and call the app to delete all its boxes before deleting the app itself).

Now that we understand some basics about how we can interact with storage on Algorand, let’s discuss two key mechanics mentioned above: foreign arrays and minimum balance requirements. These both exist to guarantee performance of the blockchain and are a form of “payment” for work and storage on-chain.

Paying for Access to State: Reference Arrays in App Calls

Everything on the blockchain is controlled by transactions. When dealing with apps, the main transaction type is an app call transaction (there are separate transaction types for creating and deleting apps, but those are not relevant here).

As part of an app call, the caller can specify what to include in the smart contract reference arrays: the apps array, accounts array, assets array, and now boxes array. This describes the set of objects that this app call will need to interact with (read and/or write, or send transactions to). These reference arrays are limited to 8 total objects per app call transaction.

For the box array, references are shared across the atomic transaction group. For a certain app to have access to manipulate one of its boxes (a box that it created or is attempting to create), the box reference can be in any of the box arrays in the transaction group.

Box Array Details & Examples

The box array is an array of pairs: the first element of each pair is an integer specifying the index into the foreign application array, and the second element is the key name of the box to be accessed.

Each entry in the box array allows access to only 1kb of data. For example, if a box is sized to 4kb, the transaction must use four entries in this array. To claim an allotted entry a corresponding app Id and box name need to be added to the box ref array. If you need more than the 1kb associated with that specific box name, you can either specify the box ref entry more than once or, preferably, add “empty” box refs [0,””] into the array. If you specify 0 as the app Id the box ref is for the application being called.

For example, suppose the contract needs to read “BoxA” which is 1.5kb, and “Box B” which is 2.5kb, this would require four entries in the box ref array and would look something like:

boxes=[[0, "BoxA"],[0,"BoxB"], [0,""],[0,""]]

The box reference budget is based on the sizes of the boxes accessed, not the amount of data read or written. For example, if a contract accesses “Box A” with a size of 2kb and “Box B” with a size of 10 bytes, this requires both boxes be in the box reference array and one additional reference, which should be an “empty” box reference.

Access budgets are summed across multiple application calls in the same transaction group. For example in a group of two smart contract calls, there is room for 16 array entries (8 per app call), allowing access to 16kb of data. If an application needs to access a 16kb box named “Box A”, it will need to be grouped with one additional application call and the box reference array for each transaction in the group should look similar to this:

Transaction 0: [0,”Box A”],[0,””],[0,””],[0,””],[0,””],[0,””],[0,””],[0,””]
Transaction 1: [0,””],[0,””],[0,””],[0,””],[0,””],[0,””],[0,””],[0,””]

Box refs can be added to the boxes array using goal or any of the SDKs.

goal app method --app-id=53 --method="add_member2()void" --box="53,str:BoxA" --from=CONP4XZSXVZYA7PGYH7426OCAROGQPBTWBUD2334KPEAZIHY7ZRR653AFY

# Algorand Python SDK
# Using ATC
atc = AtomicTransactionComposer()
atc.add_method_call(app_id, my_method, addr, sp, signer, method_args=[1,5], boxes=[[app_id, “key”]],
)

#Beaker framework
 result = app_client.call(
    Myapp.my_method,
boxes=[[app_client.app_id, "key"]],
 )

Paying for Persistent State: Minimum Balance Requirements

Foreign arrays (covered above) ensure that you’re paying for state manipulation operations. A different concept is needed to limit use of persistent storage (data in the blockchain ledger). At Algorand we use the minimum balance requirement mechanic.

In short, for any persistent piece of state, be it an account, an asset, an app, or a box, an amount of Algo proportional to the amount of storage is “locked” in a certain account. In other words, that account now has a minimum balance requirement (MBR).

• Accounts have an MBR
• Creating or owning an asset entails an increase in MBR
• Creating or opting in to an app entails an increase in MBR
• An app account’s MBR increases for each box it creates.

Box MBR Details & Example

When a box with name n and size s is created, the MBR is raised by 2500 + 400 * (len(n)+s) microAlgos. When the box is destroyed, the minimum balance requirement is decremented by the same amount.

Notice that the key (name) is included as part of the MBR calculation.

For example, if a box is created with the name “BoxA” (a 4 byte long key) and with a size of 1024 bytes, the MBR for the app account increases by 413,700 microAlgos:

(2500 per box) + (400 * (box size + key size))
(2500) + (400 * (1024+4)) = 413,700 microAlgos

Usage of Boxes

Boxes are useful in many scenarios:

• Applications that need larger or unbound contract storage.
• Applications that want to store data per user, but do not wish to require users to opt-in to the contract.
• Applications that have dynamic storage requirements.
• Applications that require larger storage blocks that can not fit in the existing global state key-value pairs.
• Applications that require storing arbitrary maps or hash tables.

The AVM supports many new opcodes that allow apps to manipulate boxes. The TEAL opcodes and the associated PyTeal are covered in the following sections.

Creating a Box

The AVM supports two opcodes box_create and box_put that can be used to create a box.
The box_create opcode takes two parameters, the name and the size in bytes for the created box. The box_put opcode takes two parameters as well. The first parameter is the name and the second is a byte array to write. Because the AVM limits byte arrays to 4,096, box_put can only be used for boxes with length <= 4,096.

// 100 byte box created with box_create
byte “Mykey”
int 100
box_create
….
// create with a box_put
byte "Mykey"
byte “My data values”
box_put

# 100 byte box created with box_create
App.box_create(“MyKey”,Int(100))
…
# box created with box_put
App.box_put(“MyKey”, “My data values”)

Box names must be unique within an application. If using box_create, and an existing box name is passed with a different size, the creation will fail. If an existing box name is used with the existing size, the call will return a 0 without modifying the box contents. When creating a new box the call will return a 1. When using box_put with an existing key name, the put will fail if the size of the second argument (data array) is different from the original box size.

Writing to a Box

The AVM provides two opcodes, box_put and box_replace, to write data to a box. The box_put opcode is described in the previous section. The box_replace opcode takes three parameters, the key name, the starting location and replacement bytes.

byte “MyKey” 
int 10
byte “best”
box_replace

   #Beaker 
    @external
    def replace_string(self, ky: abi.String, start: abi.Uint64, replacement: abi.String, *, output: abi.String):
        return Seq(
            App.box_replace(ky.get(), start.get(), replacement.get()),
            boxstr :=  App.box_get(ky.get()),
            Assert( boxstr.hasValue()),
            output.set(boxstr.value()),
        ) 

When using box_replace, the box size can not increase. This means if the replacement bytes, when added to the start byte location, exceed the upper bounds of the box, the call will fail.

Reading from a Box

The AVM provides two opcodes for reading the contents of a box, box_get and box_extract. The box_get opcode takes one parameter which is the key name for the box. It reads the entire contents of a box. The box_get opcode returns two values. The top-of-stack is an integer that has the value of 1 or 0. A value of 1 means that the box was found and read. A value of 0 means that the box was not found. The next stack element contains the bytes read if the box exists, else it contains an empty byte array. box_get fails if the box length exceeds 4,096.

byte “MyKey”
box_get
assert //verify that the read occurred and we have a value
//box contents at the top of the stack

        return Seq(
            App.box_put(Bytes("Cnt"), val.encode()),
            boxint :=  App.box_get(Bytes("Cnt")),
            Assert(boxint.hasValue()),
            output.decode(boxint.value()),
        )

Note that when using either opcode to read the contents of a box, the AVM is limited to reading no more than 4kb at a time. This is because the stack is limited to 4kb entries. For larger boxes, the box_extract opcode should be used to perform multiple reads to retrieve the entire contents.

The box_extract opcode requires three parameters: the box key name, the starting location, and the length to read. If the box is not found or if the read exceeds the boundaries of the box the opcode will fail.

byte “BoxA”
byte “this is a test of a very very very very long string”
box_put
byte “BoxA”
int 5
int 9
box_extract
byte “is a test”
==
assert

        return Seq(
            App.box_put(Bytes("BoxA"), Bytes("this is a test of a very very very very long string")),
            output.set(App.box_extract(Bytes("BoxA"), Int(5), Int(9))),
        ) 

Getting a Box Length

The AVM offers the box_len opcode to retrieve the length of a box. This opcode can also be used to verify the existence of a particular box. The opcode takes the box key name and returns two unsigned integers (uint64). The top-of-stack is either a 0 or 1, where 1 indicates the existence of the box and 0 indicates the box does not exist. The next is the length of the box if it exists, else it is 0.

byte “BoxA”
byte “this is a test of a very very very very long string”
box_put
byte “BoxA”
box_len
assert
int 51
==
assert

        return Seq(
            App.box_put(Bytes("BoxA"), Bytes("this is a test of a very very very very long string")),
            bt := App.box_length(Bytes("BoxA")),
            Assert(bt.hasValue()),
            output.set(bt.value()),
        ) 

Deleting a Box

The AVM offers the box_del opcode to delete a box. This opcode takes the box key name. The opcode returns one unsigned integer (uint64) with a value of 0 or 1. A value of 1 indicates the box existed and was deleted. A value of 0 indicates the box did not exist.

byte ”BoxA"
byte “this is a test of a very very very very long string”
box_put
byte “BoxA”
box_del
bnz existed

        return Seq(
            App.box_put(Bytes("BoxA"), Bytes("this is a test of a very very very very long string")),
            output.set(App.box_delete(Bytes("BoxA"))),
        ) 

Example: Storing Named Tuples in a Box

If your contract is using the ABI and authored in PyTeaI, you might want to store a named tuple in a Box. It is preferable that the tuple only contain static data types, as that will allow easy indexing into the box. The following example creates a box for every address that calls the contract’s add_member method. This is an effective way of storing data for every user of the contract without having to have the user’s account opt-in to the contract.

# This example uses the Beaker framework

from algosdk import *
from pyteal import *
from beaker import *


class NamedTupleBox(Application):

    class MembershipRecord(abi.NamedTuple):
        role: abi.Field[abi.Uint8]
        voted: abi.Field[abi.Bool]


    @external
    def add_member(self, role: abi.Uint8, voted: abi.Bool,*, output: MembershipRecord):
        return Seq(
            output.set(role, voted),
            App.box_put(Txn.sender(), output.encode()),
        )

    @external
    def del_member(self,*, output: abi.Uint64):
        return Seq(
            output.set(App.box_delete(Txn.sender())),
        )     

if __name__ == "__main__":
    accts = sandbox.get_accounts()
    acct = accts.pop()


    app_client = client.ApplicationClient(
        sandbox.get_algod_client(), NamedTupleBox(), signer=acct.signer
    )

    app_client.create()
    app_client.fund(100 * consts.algo)
    print("APP ID")
    print(app_client.app_id)
    print(acct.address)
    ls = acct.address.encode()

    result = app_client.call(
        NamedTupleBox.add_member,
        role=2,
        voted=False,
        boxes=[[app_client.app_id, encoding.decode_address(acct.address)]],
    )
    result = app_client.call(
        NamedTupleBox.del_member,
        boxes=[[app_client.app_id, encoding.decode_address(acct.address)]],
    )    

    print(result.return_value)
    NamedTupleBox().dump('./artifacts')

Final Noteworthy Details

Boxes store the box key and box data as byte arrays. Since the AVM stack supports both byte arrays and unsigned integers (uint64) values, it is necessary to convert uint64 variables to byte arrays before trying to store them.

Boxes may only be accessed (whether reading or writing) in a Smart Contract’s approval program, not in a clear state program.

Boxes can only be manipulated by the smart contract that owns them. While the SDKs and goal cmd tool allow these boxes to be read off-chain, only the smart contract that owns them can read or manipulate them on-chain.

Boxes can be created and deleted, but once created, they cannot be resized. At creation time, boxes are filled with 0 bytes up to their requested size. The box’s contents can be changed, but the size is fixed at that point. If a box needs to be resized, the box first must be deleted and then recreated with the new size.