IPFS Pinning With Crust

IPFS

Algorand offers various ways to store data in contracts, but there are still many use cases where storing the data off-chain makes more sense. This is paticularly true when the data is large and not used directly on-chain (for example, NFT metadata and images). A common solution for off-chain data storage is the InterPlanetary File System (IPFS) protocol. In short, IPFS is a peer-to-peer file sharing protocol. For more information on IPFS, see https://docs.ipfs.tech/concepts/faq/.

In order to share files via IPFS, one must pin a file on the network. Pinning a file means assigning it a unique Content Identifier (CID) and making it availible to download. It is common for developers to use a pinning service like Pinata, web3.storage, or nft.storage. While these services do indeed pin the file on IPFS, they are still using centralized servers to do so. This means those using these services are dependend on them to keep running them and are locked into their pricing model.

Crust

To avoid using centralized services for IPFS pinning, Algorand developers can use the Crust network. Crust is a decentralized pinning service where users can pay the network to pin a file and that file will be pinned on many servers around the world. The pricing model is set by the node runners, rather than a single entity. For more information on Crust, see https://crust.network/faq/.

Crust and Algorand

Crust is easier than ever to use for Algorand developers because you can pay for storage via ABI method calls to the Crust contracts deployed on testnet and mainnet.

Deployments

Testnet storage contract application ID: 507867511

Mainnet storage contract application ID: 1275319623

Usage

The easiest way to use the Crust storage contract is to use the ARC32 application.json that was generated by the beaker contract. The JSON and full source can be found at https://github.com/crustio/algorand-storage-contract.

The general process is:

1. Build web3 authentication header
2. Upload files to IPFS
3. Get storage price
4. Place storage order

Building Header

/**
 * Generate a web3 auth header from an Algorand account
 */
function getAuthHeader(account: algosdk.Account) {
    const sk32 = account.sk.slice(0, 32)
    const signingKey = nacl.sign.keyPair.fromSeed(sk32)

    const signature = nacl.sign(Buffer.from(account.addr), signingKey.secretKey)
    const sigHex = Buffer.from(signature).toString('hex').slice(0, 128)
    const authStr = `sub-${account.addr}:0x${sigHex}`

    return Buffer.from(authStr).toString('base64')
}

Upload to IPFS

/**
 * upload a file to IPFS and get its CID and size
 * 
 * @param account Account to use to generate the auth header
 */
async function uploadToIPFS(account: algosdk.Account) {
    // Note: Not all gateways require this header
    const headers = {
        "Authorization": `Basic ${getAuthHeader(account)}`
    }

    // list of API hosts
    // https://github.com/crustio/crust-apps/blob/master/packages/apps-config/src/ipfs-gateway-endpoints/index.ts
    const apiEndpoint = 'https://gw-seattle.crustcloud.io:443/api/v0/add'

    // If you're in browser, you should be able to just use a file directly
    const formData = new FormData();
    formData.append('README.md', fs.createReadStream('./README.md'));


    const res = await axios.post(apiEndpoint, formData, {
        headers: {
            ...headers,
            // formData.getHeaders() is only required if you're using nodejs
            ...formData.getHeaders()
        }
    });

    const json: { Hash: string, Size: number } = await res.data

    return { cid: json.Hash, size: Number(json.Size) }
}

Get Storage Price

/**
 * Gets the required price to store a file of a given size
 * 
 * @param algod Algod client to use to simulate the ABI method call
 * @param appClient App client to use to compose the ABI method call
 * @param size Size of the file
 * @param isPermanent Whether the file should be added to the renewal pool
 * @returns Price, in uALGO, to store the file
 */
async function getPrice(algod: algosdk.Algodv2, appClient: StorageOrderClient, size: number, isPermanent: boolean = false) {
    const result = await (await appClient.compose().getPrice({ size, is_permanent: isPermanent }).atc()).simulate(algod)

    return result.methodResults[0].returnValue?.valueOf() as number
}

Place Order

/**
 * Uses simulate to get a random order node from the storage contract
 * 
 * @param algod Algod client to use to simulate the ABI method call
 * @param appClient The app client to use to compose the ABI method call
 * @returns Address of the order node
 */
async function getOrderNode(algod: algosdk.Algodv2, appClient: StorageOrderClient) {
    return (await (await appClient.compose().getRandomOrderNode({}, { boxes: [new Uint8Array(Buffer.from('nodes'))] }).atc()).simulate(algod)).methodResults[0].returnValue?.valueOf() as string
}

/**
 * Places a storage order for a CID
 * 
 * @param algod Algod client used to get transaction params
 * @param appClient App client used to call the storage app
 * @param account Account used to send the transactions
 * @param cid CID of the file
 * @param size Size of the file
 * @param price Price, in uALGO, to store the file
 * @param isPermanent Whether the file should be added to the renewal pool
 */
async function placeOrder(
    algod: algosdk.Algodv2,
    appClient: StorageOrderClient,
    account: algosdk.Account,
    cid: string,
    size: number,
    price: number,
    isPermanent: boolean
) {
    const merchant = await getOrderNode(algod, appClient)
    const seed = algosdk.makePaymentTxnWithSuggestedParamsFromObject({
        from: account.addr,
        to: (await appClient.appClient.getAppReference()).appAddress,
        amount: price,
        suggestedParams: await algod.getTransactionParams().do(),
    });

    appClient.placeOrder({ seed, cid, size, is_permanent: isPermanent, merchant })
}

Full Examples

To see the full scripts that use the above functions go to https://github.com/algorand-devrel/crust-examples