The State of NeoFS – January 2024

NeoFS, developed by NeoSPCC, represents a significant stride in the field of decentralized storage solutions. As an integral part of the Neo blockchain ecosystem, NeoFS offers a distributed and decentralized object storage system designed to meet the growing demand for scalable and secure data storage in the blockchain space.

This article provides a comprehensive status update on NeoFS, delving into its architecture, smart contract integration, node operation, and available resources for developers. Readers can jump to the following sections below:

• Architecture
• Smart Contract Integration
• Storage Nodes
• Developer Resources
• Live Applications

NeoFS architecture

Network map and Inner Ring

At the core of the NeoFS architecture is the Network Map, maintained by Inner Ring nodes, the validators of the NeoFS sidechain. These nodes continuously monitor the status of NeoFS storage nodes and update the Network Map accordingly. The Network Map is a multi-dimensional graph where nodes are categorized based on attributes and values. This organization is crucial for implementing a specific data placement function, which is used to determine which nodes are most appropriate for storing or retrieving data, depending on the needs of NeoFS users.

One of the notable advantages of this process is that it eliminates the need for centralized metadata storage, commonly used to track objects in other distributed systems. As an additional advantage, this reduces the need for data rebalancing when new nodes join and leave the network, as often found in Distributed Hash Table (DHT) based systems.

Containers and Storage Policies

In NeoFS, object data is organized into containers, analogous to directories in a file system. Containers are served by a subset of storage nodes, corresponding to the requirements laid down in its accompanying storage policy. There is no hard limit on the size of an object, as large objects are broken into smaller parts which are linked and can be recombined when the original object is required.

Storage nodes collectively ensure that the storage policy is adhered to and that the data remains free from corruption. Compliance is validated using a two-stage auditing game which can prove with very high confidence that a node is truly storing the expected objects.

NeoFS sidechain

All internal operations for NeoFS, such as banking and audit results, are recorded on a dedicated sidechain to reduce load on the Neo N3 MainNet. This sidechain is managed by the Inner Ring nodes via a set of sidechain-specific smart contracts.

NeoFS platform architecture (Source: Neo SPCC)

A NeoFS entrypoint smart contract is also deployed on the Neo N3 MainNet, providing a bridge for users to move GAS tokens between MainNet and their NeoFS sidechain accounts. This arrangement also ensures the anonymity of Inner Ring nodes and prevents disclosure to other network nodes.

Protocol gateways

Another defining feature of the NeoFS architecture is its use of protocol gateways. The HTTP and S3 gateways allow existing applications to easily integrate with data stored on NeoFS. More recently, NeoSPCC added the REST gateway in a bid to simplify the onboarding process for web developers.

Creation of the REST gateway was facilitated by support for WalletConnect in the API, a milestone improvement that offers a practical signing solution to dApp developers. It is expected to replace the HTTP gateway as the primary choice for those building NeoFS-based experiences.

The team has so far provided two example applications which make use of protocol gateways. Send.NeoFS uses the HTTP gateway to serve an object upload service, and Panel.NeoFS provides an account management and container creation service using the REST gateway.

Smart contract integration

Though powerful as a standalone tool, one of the most noteworthy elements of NeoFS is its tight integration with the Neo N3 blockchain network. Neo smart contracts can directly request NeoFS data using the native oracle service. To a contract author, the request is as simple as any other oracle request to an external website. This can provide a path to fully decentralized applications, or allow builders to reduce their operational costs and make business data verifiable.

As part of the Neo APAC hackathon, the Neo SPCC team hosted a live workshop, demonstrating how applications may integrate with NeoFS in different ways. The live stream begins with some key concepts, including container creation, how to check NeoFS GAS balances, and object uploading. This is followed by an example Go smart contract which uses the oracle service to request the object previously uploaded to NeoFS and place it into contract storage.

It should be noted that the native oracle can currently only be used to retrieve data from NeoFS, not upload it to the network. This limitation remains an open discussion, support for PUT or POST has the potential to emerge in the future which could offer developers increased flexibility.

Though the demonstration was geared towards Go developers, who build contracts using the NeoGo compiler, the fundamentals demonstrated in the workshop can be replicated with any supported language.

Storage nodes

Data stored on NeoFS is hosted by storage nodes. The appropriate set of nodes to use to store data for a particular container is calculated deterministically. This is achieved by filtering the Network Map with a data placement rule, which is provided by the user during container creation. For example, a user may specify a desired geographical location, storage type, or capacity requirements. For equivalent nodes that meet the criteria, the placement rule defaults to the cheaper price setting.

Storage nodes are also responsible for controlling access to a given piece of data. Users can leverage a basic or advanced access control list to specify read and write permissions as required.

Storage nodes serve data through gateways and are synchronized via the blockchain (Source: Neo SPCC)

The cost of storing data on NeoFS has two parts. There is a standard network-wide basic income, where nodes are rewarded for available objects according to a committee-controlled basic income rate. Additionally, a node can set its own fee for data auditing. These payments are triggered when a user creates a storage group object in a container that the node is serving.

At the time of press, the basic income rate is set at 0.0000001 GAS per GB per epoch. One epoch is currently set at 240 blocks, or approximately 3600 seconds.

To guide users through storage node deployment, Neo SPCC has provided a Docker-based setup guide. A set of Ansible roles for different NeoFS node types has also been made available. Ansible is an automation tool that makes it easy to deploy and configure software. The provided playbooks can be used to operate protocol gateways, Inner Ring, and storage nodes.

The team is in the process of exploring additional simplifications, intended to make storage node operation more approachable in the future.

Developer resources

The Neo SPCC GitHub page provides a readme which serves as a landing page, introducing the team’s core offerings and developer resources.

Links can be found to the NeoFS technical specification, which details key concepts, as well as tools for builders such as the Go SDK, NeoFS API, all-in-one private network solution, and NeoFS Gateways. A NeoFS development environment is also available for use, though it is intended for the NeoFS development team, with NeoFS integrators generally better served by the AIO solution.

The NeoFS status page is maintained by the team, providing useful insights for both MainNet and TestNet. Developers will find a network health check with epoch and container counters, NeoFS smart contract identifiers, and some public gateways for testing. The page lists public nodes, including storage nodes and RPC nodes for both the Neo network and NeoFS sidechain. Storage node locations are also shared on a world map.

For one-on-one assistance, various Neo SPCC team members frequent the official Neo Discord. Members with the Builder role (from the #🧱│builder channel) can reach out for support in the dedicated NeoFS channel.

Live applications

Several practical applications have begun leveraging the benefits of NeoFS. These include in-house offerings from the development team itself and third-party integrators.

Example integrations

While the Send and Panel example apps demonstrate NeoFS utility to regular users, Neo SPCC maintains various integrations with developers in mind.

Forks of the popular restic and rclone tools for cloud backups and file copying are provided, each with NeoFS support added, and the team hosts both its official team website and the neo.org NeoFS subdomain on NeoFS. All website contents are stored inside a NeoFS container, which is served via an HTTP gateway.

The team also uses NeoFS to store Continuous Integration (CI) reports, which are automatically uploaded via GitHub actions and used to further battle-test the infrastructure. To help other teams adopt NeoFS for similar purposes, a repository has been created which provides an example GitHub action for pushing data to NeoFS.

Production applications

Several early adopters have begun using NeoFS in the real world. Most of these integrations leverage the data storage capabilities. Games such as World of Elements and Pixudi started using NeoFS to host and distribute game assets, and various NFT collections opted to store image data on NeoFS, spearheaded by the efforts of marketplaces such as SkyHut and MegaOasis in facilitating the deployment.

Dropbox-inspired interface for NeoFS users (Source: Greenfinch)

The most notable NeoFS integration can be found in Greenfinch, which provides a user-friendly, Dropbox-style frontend interface to the platform. The tool guides users through the process of loading their NeoFS balance, creating containers, uploading objects, and sharing publicly accessible links via an HTTP gateway.

Through these applications, Neo SPCC not only validates the practical use of NeoFS but also illustrates the broad range of possibilities it offers for different sectors, from content hosting to gaming and digital art. These live applications serve as real-world examples of how NeoFS’s decentralized storage solution can be effectively utilized across various industries, paving the way for more widespread adoption and innovative use cases in the future.

Source