This content originally appeared on DEV Community and was authored by Hedera Team
By Mica Cerone & Mark Blackman
Hedera is dedicated to maintaining network stability while ensuring fair and permissionless availability for all users. One way that Hedera achieves this is by using throttles, which regulate the rate at which operations (transactions and queries) are processed, both across the network and locally at individual nodes.
This blog offers a comprehensive technical overview of Hedera's throttles, explaining their purpose, functionality, and configuration.
Why Have Throttles?
Throttles regulate the rate of processing operations to ensure network stability and availability. They safeguard the network from being overwhelmed by excessive traffic or targeted by Denial of Service (DoS) attacks.
Hedera’s throttles serve to:
- Prevent overloading of the network or individual nodes.
- Ensure fair resource usage across all users and operations.
- Maintain predictable transaction processing availability and performance.
- Control the growth of the state size, ensuring long-term scalability and cost efficiency for the network.
Hedera’s throttles are configured via a throttles.json file within the Hedera Services repo. An example of a Hedera mainnet throttle can be found here. Throughout this blog, we’ll reference the syntax of this JSON file to illustrate how throttles are defined and applied.
Leaky Bucket
Hedera's throttling system is conceptually modeled after a bucket with a leak at the bottom. The total volume of the bucket represents its maximum capacity; as water (representing operations) drains through the leak, more capacity becomes available for new operations. If the bucket is full, operations are throttled by the bucket.
More specifically, a bucket is a logical container that groups specific operations and enforces a limit on the rate at which those operations can be processed—known as a rate limit. The rate limit defines the average number of operations per second) that the bucket can sustain over time. This steady-state rate reflects how consistently the bucket "drains," dictating how frequently new operations can be accepted.
In the throttles.json file, rate limits can be expressed as:
- opsPerSec: Specifies the transactions per second as a whole number.
- milliOpsPerSec: Specifies a finer-grained rate by dividing the value by 1,000.
When an operation is rejected by a throttle, the system will respond with a status code of BUSY.
Example Configuration
Here’s a bucket configuration example, simplified for understanding:
{
"name": "XYZ",
"opsPerSec": 10,
"milliOpsPerSec": 0,
"operations": [
"FileGetContents",
"FileGetInfo",
"ContractGetInfo"
]
}
From this configuration:
- The bucket name is XYZ.
- The rate limit is set to 10 operations per second (no finer rate is specified since milliOpsPerSec is 0).
- The bucket applies to three Hedera operations: FileGetContents, FileGetInfo, and ContractGetInfo.
The XYZ bucket can support any combination of these FileGetContents, FileGetInfo, and ContractGetInfo operations, but only at a sustained rate of 10 operations per second.
Network vs. Node Throttles
It’s important to understand that a bucket’s rate limit is applied at both the network and node levels. The values specified in throttles.json represent the network-wide rate limit for each bucket.
For example, in the XYZ bucket configuration:
- The network-wide rate limit supports 10 operations per second for FileGetContents, FileGetInfo, and ContractGetInfo.
At the node level, the throttle is calculated by dividing the network-wide rate limit by the total number of active nodes in the network. For instance, if there are 5 nodes in the network:
This ensures that each node can independently process these operations at a sustained rate of 2 operations per second.
Network and node throttles serve distinct but complementary purposes in Hedera's system:
-
Network Throttles:
- Manage the total transaction capacity across all nodes to ensure system-wide stability and prevent overloading
-
Node Throttles:
- Divide the network's capacity equitably among nodes, ensuring no single node monopolizes system capacity
Throttle Groups
Buckets in Hedera’s throttling system can contain one or more throttle groups, allowing a bucket to enforce multiple rate limits across different sets of operations. This flexibility ensures that specific types of operations are throttled independently, even when they share the capacity of a bucket.
Throttle groups allow fine-tuned rate limits across different transaction types within a bucket. Also, by separating operations into groups, resources can be distributed more effectively to meet network needs.
Here’s an example of a bucket with multiple throttle groups:
{
"name": "ABC",
"throttleGroups": [
{
"opsPerSec": 0,
"milliOpsPerSec": 2000,
"operations": [
"CryptoCreate",
"NodeCreate"
]
},
{
"opsPerSec": 0,
"milliOpsPerSec": 5000,
"operations": [
"ConsensusCreateTopic"
]
},
{
"opsPerSec": 0,
"milliOpsPerSec": 100000,
"operations": [
"TokenCreate",
"TokenAirdrop"
]
}
]
}
Understanding the Configuration
- Bucket Name: ABC.
-
Throttle Groups:
-
Group 1:
- Operations: CryptoCreate, NodeCreate
- Rate Limit: 2 operations per second (2000 milliOpsPerSec ÷ 1000 = 2 opsPerSec).
-
Group 2:
- Operations: ConsensusCreateTopic
- Rate Limit: 5 operations per second (5000 milliOpsPerSec ÷ 1000 = 5 opsPerSec).
-
Group 3:
- Operations: TokenCreate, TokenAirdrop
- Rate Limit: 100 operations per second (100000 milliOpsPerSec ÷ 1000 = 100 opsPerSec).
-
Group 1:
Each throttle group has its own independent rate limit but shares the overall capacity of the bucket. When operations enter the system and are associated with a specific throttle group, the operation is applied to the bucket’s capacity at the rate limit defined in the throttle group.
Example Usage of Bucket ABC
If the ABC bucket is initially empty, it can service any combination of operations within the defined rate limits. Here are a few examples of valid combinations of operations:
- 2 CryptoCreate (Group 1).
- 1 NodeCreate and 1 CryptoCreate (Group 1).
- 5 ConsensusCreateTopic (Group 2).
- 50 TokenCreate and 1 CryptoCreate (Groups 3 and 1).
- 100 TokenAirdrop (Group 3).
Multi-bucket Configuration
While throttle groups offer fine-grained control over resource allocation, relying on a single bucket introduces the risk that one specific operation could monopolize system resources. This could lead to unfair usage and reduced availability for other operations. By implementing a multi-bucket configuration, you can ensure a fair distribution of resources and maintain availability for various operations.
In a multi-bucket setup, an incoming operation is applied to one or more bucket(s) that include it in a throttle group. The operation is accepted only if all associated buckets have sufficient capacity. If any of the buckets are full, the transaction is rejected.
This approach allows for more sophisticated throttling, ensuring no single operation can overwhelm the system.
Burst Period
As mentioned earlier, the rate limit represents the average number of operations per second, reflecting the steady-state capacity of the network. However, network usage is rarely consistent and often fluctuates due to periods of high or low activity.
The burst period feature allows nodes to temporarily exceed their steady-state rate limit for a defined period. This means multiple seconds’ worth of throttle capacity can be utilized in a single second. The system continuously tracks usage to ensure that, even with burst periods, the average number of operations per second remains aligned with the configured rate limit over time.
The burst period is particularly valuable for throttles with very low rate limits, especially when the per-node capacity is less than one transaction per second. By configuring a burst period, nodes can execute these low-frequency operations during the designated burst window, ensuring they are processed periodically while still adhering to the long-term average rate limit.
Here’s an example of a burst period configuration:
{
"burstPeriod": 0,
"burstPeriodMs": 15000,
"name": "123",
"throttleGroups": [
{
"opsPerSec": 0,
"milliOpsPerSec": 2000,
"operations": [
"CryptoCreate",
"NodeCreate"
]
}
]
}
For this configuration, we see:
- Bucket Name: 123
- Throttle Group: Contains operations for CryptoCreate and NodeCreate.
- Rate Limit: The throttle group has a rate limit of 2 operations per second (2000 ÷ 1000).
- Burst Period: Configured for 15 seconds (15000 ÷ 1000). This value can be defined in seconds (burstPeriod) or milliseconds (burstPeriodMs).
If this configuration is applied to a 10-node network, here’s the impact that the burst period would have at the node level:
- The per-node rate limit would be 0.2 operations per second (2 operations per second ÷ 10 nodes).
- During the burst period, each node could accumulate throttle capacity for up to 15 seconds at the steady-state rate.
- This allows a node to process a burst of 3 operations during the burst period (15 seconds × 0.2 operations per second = 3 operations).
EVM Throttles
For EVM-related operations, an additional throttle mechanism is used. Rather than throttling by operations per second, EVM throttles are expressed in gas per second. This better reflects the consumption of resources performed to execute the operation. Specifically, ContractCall, ContractCallLocal, and ContractCreate operations are subject to these throttles.
EVM related operations include two separate throttles:
- Frontend Throttle: Regulates the maximum gas per second for incoming gRPC transactions at the node level. The transaction is rejected with a status code of BUSY.
- Consensus Throttle: Enforces the maximum gas per second during the execution of transactions at the Hedera consensus level. It is analogous to Ethereum’s Block Gas Limit. The transaction is rejected with a status code of CONSENSUS_GAS_EXHAUSTED. EVM throttles are not configured through throttles.json; instead, they are configured using two global dynamic properties configured from Hedera file 0.0.121:
- contracts.frontendThrottleMaxGasLimit: Defines the frontend throttle limit for maximum gas per second.
- contracts.consensusThrottleMaxGasLimit: Specifies the consensus throttle limit for maximum gas per second.
Please note: Contract-related transactions can still be subject to the standard opsPerSec buckets where contract-related operations are listed. These buckets operate independently and may reject transactions based on the configured operations-per-second limits.
Conclusion
Hedera’s throttling system is vital to its network infrastructure, ensuring stability, fairness, and efficient resource utilization across diverse operations. As the network grows and evolves, we anticipate that system throttling will continue to adapt to better safeguard critical resources while maintaining full availability and optimal performance for users.
This content originally appeared on DEV Community and was authored by Hedera Team
Hedera Team | Sciencx (2025-02-10T17:39:30+00:00) Throttling in Hedera: Ensuring Stability and Fairness. Retrieved from https://www.scien.cx/2025/02/10/throttling-in-hedera-ensuring-stability-and-fairness/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.