Substrate Runtime Developer Academy Course
Last updated
Was this helpful?
Last updated
Was this helpful?
Understand the Polkadot ecosystem on Substrate Codebase. Learn blockchain design principles. Learn Polkadot.js SDK. Launch your first blockchain, and build a web3 front-end DApp.
ICP = Industry Connect Points , you gain point by asking questions or answering questions on Q-Hub (question Hub)
UseFull Links:
Rust Doc:
& ==>> the 2 big ref doc official
A cook book substrate: -> very good one
To run and build rust tool with Cargo web site:
For the course:
Substrate is a framework to build blockchain platform.
There are many program to help on new projects:
Parity Substrate Builder Program
Web3 Foundation Grant Program
Polkadot/Kusama on chain Treasury
Multiple Accelerator and Incubators
Substrate client consists of 2 parts:
Runtime:
on-chain state transition logic
Client:
All off-chain components that are able to exectue the runtime
Storage
RocksDB -> the default DB used by many blockchain projects
TrieDB -> in early stage but more optimised for Substrate
Consensus (PoW and PoS are provided)
Babe & Granpa
p2p network (communication between substrate nodes)
RPC (enables clients/wallet to interact with the blockchain)
Runtime Executor (use to validate block, produce new block, call by block authoring, or blockimport)
Edmund keys are used to sign transactions
Substrate is Designed by Web3 Foundation and developed by Parity Tech.
Polkadot Ecosystem:
Relay chain: Polkadot, Kusama. Responsabile of security, consensus and cross-chain interop
Parachain/Parathread: sovereign chain. Parachain needs to rent a slot from relaychain. Parathread nees token from Parachain to produce blocks
Bridge: allows communications with external blockchain like Ethereum, Bitcoin
3 aspects:
Performance
Innovation:
Module oriented
WASM Sandbox
FRAME is the blockchain building framework provided by substrate, for building module
ink is a new WASM SmartContract framework, so developed in Rust and deployed on a substrate blockchain
Possible to deploye Solidity SC on Substrate
Reuse of existing 3th party lib (like zcash lib)
Off-chaine worker (for oracle)
Polkadot Ecosystem:
a network of purpose build blockchain
internet of blockchain
public permissionless blockchain, developped by Web3 and Parity
PoC 2 is the predecessor of Substrate
Relay Chain: Kusama, Polkadot
Reponsible of security, consensus, parachain
Parachain: independant blockchain, connected to relaychain
Parathread -> pay on usage: pay token on relay chain to validate blocks
Bridges: eth, btc
Role: Validators part of Relaychain, and collators node part only of the parachain.
File in a Substrate Project:
A Dispatchable call is a function that a blockchain user can call as part of an Extrinsic.
Inherents are pieces of information that are not signed and only inserted into a block by the block author. They are not gossiped on the network or stored in the transaction queue. Like timestamp added by validator on the block.
Signed transactions contain a signature of the account that issued the transaction and stands to pay a fee to have the transaction included on chain
Use unsigned transactions with care, as their validation logic can be difficult.
SignedExtension
is a trait by which a transaction can be extended with additional data or logic. Signed extensions are used anywhere you want some information about a transaction prior to execution. This is heavily used in the transaction queue.
The transaction pool contains all transactions (signed and unsigned) broadcasted to the network that have been received and validated by the local node.
Weights represent the limited time that your blockchain has to validate a block. This includes computational cycles, and storage I/O.
!!!! Keep in mind that FRAME is not the only system for Substrate runtime development.!!!
The Substrate runtime is composed with a set of primitive types that are expected by the rest of the Substrate framework.
Core Primitives:
hash
digestItem
digest
extrinsic
header
block
blocknumber
Frame Primities:
call
origin
hashing (like blake2)
accountId
events
version
All other pallets depend on the System library as the basis of your Substrate runtime.
Pallet frame exists in 2 version today (v1 and v2).
Skeleton of v1 frame: (composed of 5 sections)
Import and dependancies
runtime config traits
runtime events
runtime storage
pallet declaration (module struc exportable for that pallet)
Skeleton of v2 frame: (composed of 7 sections)
Import and dependancies
declaration of pallet types
runtime config traits
runtime storage
runtime event
hooks (to be able to call ext component from pallet)
extrinsics (functions callable from external pallet)
macros are lines of code that write code.
Overview of substrate macros:
macros in the Frame support library
marcos in the Substrate system library
Frame 1 Declarative Macro:
decl_Storage!
decl_event!
decl_error!
decl_module!
Frame 2 Declarative Macro:
...
Additional Frame Macro:
construct_runtime!: To construct Substrate runtime and integrating various pallets into the runtime.
parameter_types!: To declare parameter types to be assigned to pallet configurable trait associated types during runtime construction.
...
Substrate system library macro:
impl_runtime_apis!: This macro generates the API implementations for the client side through the RuntimeApi
andRuntimeApiImpl
struct type.
app_crypto!: To specify cryptographic key pairs and its signature algorithm that are to be managed by a pallet.
Runtime Metadata:
Runtime Storage:
Runtime storage allows you to store data in your blockchain that is persisted between blocks and can be accessed from within your runtime logic.
Blockchains that are built with Substrate expose a remote procedure call (RPC) server that can be used to query runtime storage.
You can install the following module to start using Rust on VS:
CodeLLDB
rust-analyzer (desactivate Rust officila module, make conflict)
Even Better TOML
Rust Test Explorer (View and run your Rust tests in the Sidebar of Visual Studio Code)
Install tools:
on VS Code, install dependencies:
rust-lang
bungcip.better-toml
vadimcn
Service hosted by Polkadot
Install the node-template:
To run one local node-template and the Frond-end :
Then you can transfert 10 units from Alice to Ferdie
If you want to display variable content during test, you can do the following. In the rust code, you use println!("My value is: {:#?}", my_val); . Then when you launch the test run the following. Important to add --exact
sudo cargo test --package pallet-kitties -- --nocapture --exact
Why?
Insure code is modular and abstracted
Reduce coupling between module so that they are independant
Maximise reusability
The core function: depend on interface/trait in place of specific implementation/module
The metadata is encoded using the scale codec. SCALE = simple concatenated aggregate little-endian. The pokadot.js is making the decoding automatically from the SCALE format to JSON. The metadata includes all the details for the runtime and the pallets, including:
storage
events
calls
constants
errors
After error, you have index, which are pallet description and you see the number of storage, events, calls, constants and errors they have. This correspond to what has been defined in the construct_runtime section of the lib.rs of the runtime package
In your Google Chrome Browser, you can see the JSON RPC Call behind the wood that polkadot.js uses to call local substrate node.
-> Go on Google Chrome menu: View -> developer -> inspect elements. Then go on the Network tab, select the 127.0.0.1 connection , to on message tab and you will see all the request dans their answers from the Substrate node. If you don't see traffic just push "cmd +R" key on your mac.
If you have error from the polkadot.js interface, you need to go to settings -> developer and enter there the custom type you have created, becasue polkadot.js is not aware of them.
Onchain storage is a key-value DB. Now how the key and how value are encoded is up to the substrate to do the work.
You can see code by going to substrate->primities->io, check the storage trait (io::storage), you can fine the following primitive methods:
get/set/clear/exists
start_transaction/rollback_transaction/commit_transaction
clear_prefix (We can delete all the key with a given prefix)
next_key
append (capability to append storage without decoding and reencoding all)
The Onchain storage format is following those rules/principles:
Storage item
Module Prefix
Storage Prefix
Storage type
Storage value
Key = twox128(module-prefix) ++ twox128(storage-prefix)
Storage map
Key = twox128(module-prefix) ++ twox128(storage-prefix) + hasher(encode(key))
Storage double map
Key = twox128(module-prefix) ++ twox128(storage-prefix) + hasher1(encode(key1)) + hasher2(encode(key2))
What are the storage hasher:
identity (no hash done here, if the hash has already be calculated before you can use this)
twox_64_concat (fast, but only to be used if input is trusted, no secure enough otherwise)
Blake2_128_concat (still fast but slover than twox, but more secure)
Deprecated storage hasher (old Kusama): twox128, twox__256, blake2__128, blake2__256
You can play with Polkadot.js following interface to query the storage of Substrate network:
Scale is the opposite of JSON, this is a lightweight format, binary format not human readable. It is not selfdescribing so you need to know the schema to decode. So it doesn't included embedded in the coded value, the info to decode it. So as developer you have to be carefull that the schema is well communicate between the rust code and the SDK.
Runtime
Frame: Framework for Runtime aggregation of modularized entities. This is a set of tools and modules to develop Substrate Runtime. In runtime modules, you have the pallets
Pallet = standardise shiping container, general name of crates/modules in rust: Trait, storage, calls, errors, events, modules, origin, banchmarks
A pallet is implemented with several macros:
decl_module
decl_storage
decl_event
decl_error
Runtime is assembled with another macro
construct_rutime
This is a pallet that does the following in essence: allows blockchain users to pay a deposit to reserve a nickname and associate it with an account they control.
If we look at the Nicks pallet in detail, we know it has:
Module Storage: Because it uses the decl_storage!
macro.
Module Events: Because it uses the decl_event!
macro. You will notice that in the case of the Nicks pallet, the Event
keyword is parameterized with respect to a type, T
; this is because at least one of the events defined by the Nicks pallet depends on a type that is configured with the Config
configuration trait.
Callable Functions: Because it has dispatchable functions in the decl_module!
macro.
The Module type from the decl_module!
macro.
Note that not all pallets will expose all of these runtime types, and some may expose more! You should always look at the documentation or source code of a pallet to determine which of these types you need to expose.
Compile the node with the new nick pallet defined
It supports all substrate base chain
Tools to generate typescript type files (to help you doing the coding)
Steps to do it:
Step1: Bump spec_version: increment by one the spec_version: go on runtime/src/lib.rs. Just increment value by one if you want the wasm runtime to be upgraded
Step2: Build the wasm file
Any time you compile a runtime, it compile with the native platform format and also compile on WASM format. The WASM portable compiled format of the node-runtime is located under the project folder in:
substrate-node-template/target/debug/wbuild/node-template-runtime/target/wasm32-unknow-unknwow/release -> node_template_runtime.wasm
Step3: use sudo to perform a runtime upgrade
On top lef, you can see the spec version number
You can also go to developer menu -> rpc -> state, getruntime() function
Select system -> setCode, select file upload and drag and drop the wasm code, then select with weigth override and put one, then submit transaction (with account Alice)
Then make the check here above to see version spec has incremented and that your new pallet are here now
Sudo is not anymore activated on Polkadot, they use the Democracy pallet.
For pallet-sudo details, please go on substrate/pallet/sudo pallet/scr/lib.rs
Off Chain worker is a innovative feature running on validator node to bridge offchain and onchain storage.
As example:
Polkadot and Kusama use offchain worker to tun Sequential Phragmen Algorithm for validator selection algorithm.
To run non deterministic tasks
Run PoW Miner
For task required HSM
Sign some data with HSM
Perform Https request to external web site and enchor onchain some result -> so implement oracle
Some technical deep dive aspect:
it is triggered everytime a block is imported (not a block is syncing)
The TreeDB used for onchain storage is equivalent to no sql storage (schemaless key-value store). Schema is defined in the code, not in the storage. It is just bunch of bytes, you don't manage data type at TreeDB level, this is the calling code that has to manage that aspect.
Automated pallet versionning is availble from version 3.X (using version in the cargo.toml file). So now unified pallet version.
Runtime Pallet:
More Flexible
You event don't need to charge transition fee for your state if you want, you can setup custom whitelist
Full access to everyting, you can call all the functions of all the other pallets.
Up to the developers to assure that storage usage can not be abused
Benchmaking to measure weight to assure that block time is reasonable
Require runtime upgrage and approval by governance, that is a big drawback -> so to deploy a new pallet you need the approval of all the participants which can take time
You need to maintiant your own bloackc
SmartContract:
limited by platform
sandbox execution environment (this put lost of complexity and limitations), so storage can be more secure
Performance penality as we need to assess each operations cost and storage usage, to calculat gas metering
State rent
Deployment can be permissionless
Pallet-Contract versus EVM(Ethereum Virtual Machine)
Pallet-Contract
WASM Bytecodes
You can use any language that compile to WASM to write your SC
You can use Rust support via ink!
AssemblyScript support via new tool ask!
Compile Solidity project to wasm using Solang project
Limited tool such as RedSpot
New ecosystem and community, all is very new
EVM
Special Bytecodes
Use Solidity or any language that compile to EVM Language
Many mature dev library and tools
Large echosystem and community
Extra tool required to be integrated with Substrate
The big difference between Eth Solidity and Substrate Rust
EVM is using 160 bits address Format why Substrate is using 256
Signing algorithm are not the same
SDK are different
RPC are different
The security is managed by the validators of the relay chain. Relay chain supports only a limited number of parachain slot. As estimation, one parachain requires 10 validators. For the moment Pokadot has 300 validators and has a program to push the number of validator to 1000, so it means about 100 parachain slots. Kusama has already 700 validators. So as the number of slot is limited, they use the parachain slot auction to lease slot on Polkadot/Kusama.
There is also a parathread approach, so pay as you go model. So you bid relay chain token for block inclusion on the relay chain. So you compete with other to insert your transaction in a relay chain block, the more you pay the bigger the change your will win the bid.
For each parachain slot, you have a set of validators node on the relay chain. Those node can communicate with the relay chain collator nodes which are full node created new block for the parachain and communicating with the relaychain validator to ask inclusing of their block in the relay chain.
The Relay chain is using a Grandpa/babe consensus and the parachain is using a cumulus-consensus:
Cumulus-Consensus: On each parachain, you run internally a relay chain node to verify, follow and finalize parachain block (you can not finalize a parachain block if the relay chain has not include it in the relay chain)
Cumulus-Runtime: Provide parachain runtime validation capabilities and proof generation routine. Parachain can validate Relay chain and vis-versa. Implement also validate_black API that is used by validator to validate parachain blocks
Cumulus-Collator: Implement parachain block producing feature
Malicious collator can not cause issue, this is really malicious valitor that can forge blocks, not the collabor.
There is a Relay-chain testnet that is Rococo for the moment. Rococo currently runs four test system parachains (Statemint, Tick, Trick, and Track), as well as several externally developed parachains
Relay-chain provides the security and validity guarantees, parachains are not subject to normal blockchain attack scenarios, like a 51% attack. Relay-chain validators will reject invalid blocks, so a parachain only needs a single honest collator to submit blocks.
Polkadot provides the security and validity guarantees, parachains are not subject to normal blockchain attack scenarios, like a 51% attack. Polkadot validators will reject invalid blocks, so a parachain only needs a single honest collator to submit blocks.
Validators use the random outputs that BABE generates — the same ones that assign block production slots — to determine which parachain to validate next. Once a validator knows its new parachain, it finds collators from that parachain to establish connections with. Each validator assigned to a parachain will import the correct state transition function to validate that parachain. Now that collators and validators share a connection and common logic, a collator can send a block to one of the validators.
By separating block production (BABE) from finality (GRANDPA), Polkadot can perform extra validity checks after a block is produced but before it is finalized.³ Polkadot has special actors, fishermen, who patrol Relay Chain blocks for invalid candidate receipts.
After enough secondary checks have been performed on all the candidate receipts within a block, validators can finally vote for that block (and by extension, all previous blocks) in GRANDPA. Once it has more than two-thirds of pre-commits, the block is in the finalized chain. All of the availability and validity checks should take place in less than one minute from the time a block is authored to the time it is finalized.
See also the following:
XCMP = Cross chain message passing:
It enables arbitrary message passing between Relaychain and parachain
This is the relay chain that garantee that message are delivered
Current dev phase is: only vertical message passing
Up ( UMP = upward message passing) : From parachain -> relaychain
Down (DMP = download message passing): From relaychain -> parachain
At this dev stage, parachain can not yet direclty send message to other parachain
HRMP (Horizontal Relay-routed message passing): With that current mecanim, parachain can send message to other parachain via relaychain -> later will be direct com parachain->parachain -> no more bottlebeck
XCM = Polkadot Cross Consensus Message Format :
Asynchronous
Absolute (message is guarantee to be delivered)
Asymmetric
Agnostic
Type of message on XCM:
WithdrawAsset
ReserveAssetDeposit (When transfering BTC to Eth)
TeleportAsset (like when transering USDT created on one platform and ported on another one)
Transact
Asset instruction:
DepositAsset
InitialeReserveWithdrawn
Remark
Multilocation (can be a blockchain, a SmartContract, a user)
MultiAsset (define token, can be a NFT-ERC721 or FT-ERC20 like)
Why to became a parachain?
Post message: !drip YOUR_ROCOCO_ADDRESS
Some issues:
Future of Polkadot:
Nested Relay Chain -> each relaychain becoms a parachain (because 100 parachain is not so big number). One nesting = 100 x 100 supported slot for parachain = 10.000 blockchains
Contribute to Echo System:
Bootstrap your project (where you can get help):
Web1.0: Static Web 1990-2000 (Read only)
Linking static content together accross the Internet
Web2.0: Dynamic Read/Write (Social Media was the big part)
Linking program to that content and building rich application accross all the devices)
Web3.0: Web2.0 + Trust (as a core layer at the foundation, we build verifiability at the core layer)
We are linking content and program direclty to each other, bypassing intermediary organisation, gaining public verifiability
In order to run the code, there is a issue in a file for WASM, the procedure to fix it is:
Solution:
When you run the code, please run the following command:
"./target/release/node-template --dev --tmp", don't forget the tmp
An extrinsic: is a piece of information that comes from outside the chain and is included in a block. Extrinsics fall into three categories: inherents, signed transactions, and unsigned transactions. "Extrinsic" is Substrate jargon meaning a call from outside of the chain. Most of the time they are transactions, and for now it is fine to think of them as transactions. Dispatchable calls are defined in the
The is responsible for accumulating the weight of each block as it gets executed and making sure that it does not exceed the limit. The is responsible for interpreting these weights and deducting fees based upon them. The weighing function is part of the runtime so it can be upgraded if needed.
The runtime of a blockchain is the business logic that defines its behavior. In Substrate-based chains, the runtime is referred to as the ""; it is where Substrate developers define the storage items that are used to represent the blockchain's as well as the that allow blockchain users to make changes to this state.
Substrate uses runtimes that are built as bytecode. Substrate also defines the that the runtime must implement.The core Substrate codebase ships with , Parity's system for Substrate runtime development. FRAME defines additional and provides a framework that makes it easy to construct a runtime by composing modules, called "pallets". Each pallet encapsulates domain-specific logic that is expressed as a set of a , , and .
Find more info on:
Substrate uses to aggregate the logic derived from pallets that are implemented for a runtime. These runtime macros allow developers to focus on runtime logic rather than encoding and decoding on-chain variables or writing extensive blocks of code to achieve .
To understand how Macro are implemented ->
Blockchains that are built on Substrate expose metadata in order to make it easier to interact with them. This metadata is separated by the different that inform your blockchain. For each module, the metadata provides information about the , , , constants, and errors that are exposed by that module. Substrate automatically generates this metadata for you and makes it available through RPC calls.
Storage info:
Substrate uses a simple key-value data store implemented as a database-backed, modified Merkle tree. All of Substrate's are built on top of this simple key-value store. Substrate implements its storage database with , a persistent key-value store for fast storage environments. It also supports an experimental .
(see )
Follow procedure here under if you have a new Mac Book Pro with M1 CPU:
Install:
Install your first Substrate-chain:
You can use the polkadot.js explorer/wallet:
Use Telemetry:
Follow the folowing steps:
The sp_IO (substrate paller Input/Output) on Rust module, check on : and the storage module:
Those are low level primites, normally we don't interact directly with them, we use
If you want to fine the timestamp of the last block, go to , then select now() method at current block and execute, you can use your Google Chrome in dev mode (Inspector, network, WSS socket, slect 127.0.0.1). You can see on the Write the query sent to Substrate and find the row storage key used to get the value. You can copy the raw key value and then select on polkadot.js in the chaine Row storage menu the value and you should find the little indian encoded version of the value.
This link is a very good deepdive on Storage :
Codec used to serialized Key and value into byte. It is also the codec used for transaction format, block format. You can find more info on : +
The SCALE (Simple Concatenated Aggregate Little-Endian) Codec is a lightweight, efficient, binary serialization and deserialization codec. It is inspired from SERD Framework () that is a Rust lib enable to serialize and deserialize many data format/structure but on a leightwheight version.
There is an Open Source Community of Opensource Runtime Module:
There is also an NFT Module used by Acala:
You can check at
In Substrate you have the notion of origin. The runtime origin is used by dispatchable functions to check where a call has come from. More info on
The acts as the orchestration layer for the runtime. It dispatches incoming extrinsic calls to the respective pallets in the runtime.
The is a collection of Rust macros, types, traits, and functions that simplify the development of Substrate pallets. The support macros expand at compile time to generate code that is used by the runtime and reduce boilerplate code for the most common components of a pallet.
Add the Nick Pallet:
Every pallet has a component called Config
that is used for configuration. This component is a ; traits in Rust are similar to interfaces in languages such as C++, Java and Go. FRAME developers must implement this trait for each pallet they would like to include in a runtime in order to configure that pallet with the parameters and types that it needs from the outer runtime.
The Nick Frame Git code:
This link is also a good example to create a pallet in its own crate: , on new version of template-node, this is Template pallet (the new test pallet)
On the FE, you will need to give them the ABI of the SC, so this is the type definition API interface part. Substrate comes with a powerfull metadata system, however it does not come with the definition of custom type. We need to provide the type definition for custom types. You can use the following interface: ()
For more information on polkadot.js api: :
You can play also with the javascript consol:
Full Substrate project is on
To see the runtime running version, go to you
Publish a transaction on the blockchain via polkadot.js tool and upload the wasm version of the new runtime. go to
Offchain worker module doc:
Pallet-Contrat is a pallet that implement WASM SC execution environement Feature:
We need to use ink!, which is a Rust contract development framework developped by Parity, for SC to be deployed on pallet-contract: , you have also:
Frontier is the pallet enabling to use existing eth tool and SC but on substrate node:
Ether.js wrapper like around polkadot.js:
Cumulus () is a set of tools to build substrate parachain if you have your own independant existing Substrate based blockchain network.
Each parachain is a chard. Each parachain is a shard of Polkadot, with unique state transition rules. Parachains have separate economies, governance mechanisms, and users. Because of the interface that Polkadot provides, the Relay Chain validators can guarantee that each parachain follows its unique rules and can pass messages between shards in a trust-free environment. Here is a very good article explaining how blocks are created, be validated in the Relay/Parachain environment:
Cumulus-Workshop:
Rococo is the test relay chain testnet:
Parachain Implementer Guidline in details: ,
XCM is format of the message used by XCMP: That draft implementation is already used in Polkadot cumulus.
See ->
See how contract pallet work
Use Faucet of Rococo to do it:
Block Explorer Rococo:
Annoying point is the metadata we have to include in polkadot.js -> bad dependency for developer consumming substrate service ()
Pallet Macro -> Move complex support Macro to attribute procedural Macro (More Std Rust):
IPFS Block protocol for querying block data ->
Transaction-less relay chain -> parachain should focus on validation parachain block and not doing other tasks
SPRE = Shared Protected Runtime Execution Enclave
go to , contribute to Issue or Pool Request
go to Open Runtime Module Library (ORML): . This is a community maintained collection of Substrate runtime modules. The substrate will have only the pallet that will be used for Polkadot and Kusama, but if you want more pallet for your project, check ORML.
PSPS (Polkadot Standard Proposal) process of Polkadot (this is like EIP with Etherum):
Substrate Open Working Groups :
Web3 foundations grants:
Web3 grants (about 30K in fiat) :
Support program if you plan to build a project on Substrate: (offer technical support)
(bounties)
Ref on :
A light Substrate version (WASM only runtime):
What bout Raspbery
Session Video 1(Dan Forbes Preso on Substrate):
Session Video 2 (MVP Tutorial):
Session Video 3:
Session Video 4:
Session Video 5:
Session Video 6:
Session Video 7:
Parachain and XCM in Depth:
See
I have clone all my exercice result on