Chunks
Chunks are native data types in the Autonomi Network:
Size: 4MB of raw bytes
Content-addressed: Address is the hash of its content
Immutable & self-verifiable: Once stored, data cannot be modified, and its integrity can be verified by computing the hash and comparing it to the address.
Client Methods
chunk_get Retrieves a chunk from the network by its address.
chunk_put Uploads a chunk to the network with payment handling. Returns the total cost and the chunk's address.
chunk_cost Estimates the storage cost for a chunk.
chunk_batch_upload Uploads multiple chunks efficiently with a single payment receipt.
Examples
Basic Chunk Operations
use autonomi::Client;
use autonomi::client::payment::PaymentOption;
use autonomi::client::chunk::{Chunk, Bytes};
use test_utils::evm::get_funded_wallet;
use eyre::Result;
#[tokio::main]
async fn main() -> Result<()> {
// Initialize a local client and test wallet
let client = Client::init_local().await?;
let wallet = get_funded_wallet();
// Create a chunk with some data
let chunk = Chunk::new(Bytes::from("Hello, world!"));
println!("Created chunk with size: {} bytes", chunk.size());
// Estimate cost before uploading
let cost = client.chunk_cost(chunk.address()).await?;
println!("Estimated storage cost: {cost}");
// Upload chunk with payment
let payment_option = PaymentOption::from(&wallet);
let (put_cost, addr) = client.chunk_put(&chunk, payment_option).await?;
assert_eq!(addr, *chunk.address());
println!("Chunk uploaded for: {put_cost} at address: {}", addr.to_hex());
// Wait for replication across the network
tokio::time::sleep(tokio::time::Duration::from_secs(5)).await;
// Retrieve and verify the chunk
let retrieved_chunk = client.chunk_get(&addr).await?;
assert_eq!(retrieved_chunk, chunk);
println!("Chunk retrieved successfully, data: {}",
String::from_utf8_lossy(retrieved_chunk.value()));
Ok(())
}Batch Upload Operations
For uploading multiple chunks efficiently:
use autonomi::{Client, client::chunk::Chunk};
use autonomi::client::payment::{PaymentOption, receipt_from_store_quotes};
use ant_protocol::storage::DataTypes;
use bytes::Bytes;
use eyre::Result;
async fn batch_chunk_upload() -> Result<()> {
let client = Client::init_local().await?;
let wallet = get_funded_wallet();
// Create multiple chunks
let data_chunks = vec![
"First chunk data",
"Second chunk data",
"Third chunk data",
];
let chunks: Vec<Chunk> = data_chunks
.into_iter()
.map(|data| Chunk::new(Bytes::from(data)))
.collect();
// Collect chunk references for batch operations
let chunk_refs: Vec<&Chunk> = chunks.iter().collect();
// Get storage quotes for all chunks
let quote = client.get_store_quotes(
DataTypes::Chunk,
chunks.iter().map(|chunk| (*chunk.address().xorname(), chunk.size()))
).await?;
// Pay for all chunks at once
wallet.pay_for_quotes(quote.payments()).await.map_err(|err| err.0)?;
let receipt = receipt_from_store_quotes(quote);
// Upload all chunks with the payment receipt
client.chunk_batch_upload(chunk_refs, &receipt).await?;
println!("Successfully uploaded {} chunks in batch", chunks.len());
// Verify all chunks are retrievable
for (i, chunk) in chunks.iter().enumerate() {
let retrieved = client.chunk_get(chunk.address()).await?;
assert_eq!(&retrieved, chunk);
println!("Verified chunk {}: ✓", i + 1);
}
Ok(())
}Error Handling
use autonomi::{Client, client::chunk::Chunk};
use autonomi::client::{GetError, PutError};
use bytes::Bytes;
use eyre::Result;
async fn chunk_error_handling() -> Result<()> {
let client = Client::init_local().await?;
let wallet = get_funded_wallet();
// Test size limits
let oversized_data = vec![0u8; Chunk::MAX_SIZE + 1];
let oversized_chunk = Chunk::new(Bytes::from(oversized_data));
if oversized_chunk.is_too_big() {
println!("Chunk exceeds maximum size of {} bytes", Chunk::MAX_SIZE);
return Ok(());
}
// Handle put errors
match client.chunk_put(&oversized_chunk, PaymentOption::from(&wallet)).await {
Ok((cost, addr)) => {
println!("Upload successful: cost={}, addr={}", cost, addr.to_hex());
}
Err(PutError::Serialization(msg)) => {
println!("Serialization error: {}", msg);
}
Err(PutError::Network { address, network_error, payment }) => {
println!("Network error at {}: {}", address, network_error);
}
Err(e) => {
println!("Other put error: {}", e);
}
}
// Handle get errors
let invalid_addr = ChunkAddress::from_hex("0000000000000000000000000000000000000000000000000000000000000000")?;
match client.chunk_get(&invalid_addr).await {
Ok(chunk) => {
println!("Retrieved chunk with {} bytes", chunk.size());
}
Err(GetError::RecordNotFound) => {
println!("Chunk not found at address: {}", invalid_addr.to_hex());
}
Err(GetError::Network(network_err)) => {
println!("Network error during retrieval: {}", network_err);
}
Err(e) => {
println!("Other get error: {}", e);
}
}
Ok(())
}Support for error handling in languages other than Rust is in development
Advanced Configuration
Performance Tuning
use std::env;
// Configure batch sizes via environment variables
fn configure_performance() {
// Set chunk upload concurrency (default: 1)
env::set_var("CHUNK_UPLOAD_BATCH_SIZE", "4");
// Set chunk download concurrency (default: 1)
env::set_var("CHUNK_DOWNLOAD_BATCH_SIZE", "8");
}Size Limits and Best Practices
Maximum chunk size: 4MB raw data (
Chunk::MAX_RAW_SIZE)Content addressing: Each chunk's address is the hash of its content
Immutability: Chunks cannot be modified once stored
Batch operations: Use
chunk_batch_upload()for multiple chunks to reduce costsCaching: Enable chunk caching for frequently accessed data
Error handling: Always handle network errors and size limit violations
Last updated