GraphOps: IaC to setup Graph Indexer Node — the Google of Blockchain

Hey Folks! I hope you are doing amazing. In this blog, I am going to write how to configure the Graph Protocol Indexer on Public Cloud using Terraform.

Challenge:

When building DApps on any blockchain, one of the problems developers face is retrieving data from the blockchain. There are hundreds and thousands of nodes to search for data. For every request made to get data from the blockchain by the DApp, a round trip is performed throughout the blockchain nodes, which is inefficient that makes applications slow.

• Some of the companies building DApps are using another workaround to retrieve data from the blockchain and store that data in a centralized location which is not good practice to do.
• To get this data, you would have to process every single transfer event ever emitted, read the metadata from IPFS using the Token ID and IPFS hash, and then aggregate it. Even for these types of relatively simple questions, it would take hours or even days for a decentralized application (Dapp) to run in a browser to get an answer.
• You could also build out your own server, process the transactions there, save them to a database, and build an API endpoint on top of it all in order to query the data. However, this option is resource intensive, needs maintenance, presents a single point of failure, and breaks important security properties required for decentralization.

Indexing blockchain data is really, really hard.

Drum roll… Please🥁🥁

Introducing The Graph 🚀

The Graph — The Graph is a decentralized protocol for indexing and querying data from blockchains, starting with Ethereum. It makes it possible to query data that is difficult to query directly.

Roles in Graph Network

• Developer — Developers build subgraphs and publish them to The Graph Network. Then, they query live subgraphs with GraphQL in order to power their applications.
• Indexer — Indexers are node operators in The Graph Network that stake Graph Tokens (GRT) in order to provide indexing and query processing services. Indexers earn query fees and indexing rewards for their services. They also earn from a Rebate Pool that is shared with all network contributors proportional to their work, following the Cobbs-Douglas Rebate Function.
• Curator — Curators are critical to the Graph decentralized economy. They use their knowledge of the web3 ecosystem to assess and signal on the subgraphs that should be indexed by The Graph Network. Through Explorer, curators are able to view network data to make signaling decisions. The Graph Network rewards curators who signal on good quality subgraphs with a share of the query fees that subgraphs generate. Curators are economically incentivized to signal early. These cues from curators are important for Indexers, who can then process or index the data from these signaled subgraphs.
• Delegator — Delegators are network participants who delegate (i.e., “stake”) GRT to one or more Indexers. Delegators contribute to securing the network without running a Graph Node themselves. By delegating to an Indexer, Delegators earn a portion of the Indexer’s query fees and rewards. The amount of queries an Indexer can process depends on the Indexer’s own (and delegated) stake and the price the Indexer charges for each query, so the more stake that is allocated to an Indexer, the more potential queries they can process.

How does the Graph Work from consumer point of view?

• Subgraph :

Subgraph define which data from a data source (such as Ethereum) will be indexed and how it is stored

The Graph learns what and how to index Ethereum data based on the subgraph description. which is also known as a subgraph manifest.

Learn more about Developing SubGraphs at Graph Academy

The flow of Graph Ecosystem

1. A Decentralised Application adds data to Ethereum through a transaction in the Smart Contract.
2. The Smart contracts emit events when processing the transaction.
3. The Graph Node continuously scans the Ethereum for new blocks and data, which we defined in the subgraph
4. Graph Node finds Ethereum events for your subgraph in these blocks and runs the mapping handlers you provided. The mapping is a WASM module that creates or updates the data entities that Graph Node stores in response to Ethereum events
5. The Decentralised application quires the Graph Node for indexed data from the blockchain, using nodes GraphQL API. The Graph node, in turn, converts the GraphQL query to internal queries to get data from the store, and the user can display the data got as a response in User Interface.

Indexers are node operators in The Graph Network

Participating in the network requires constant monitoring and interaction. The components necessary to participate in the network are

• Indexer agent — The agent monitors the network and the Indexer’s own infrastructure and manages which subgraph deployments are indexed and allocated towards on-chain and how much is allocated towards each.
• Indexer service — The only component that needs to be exposed externally, the service passes on subgraph queries to the graph node, manages state channels for query payments, and shares important decision-making information with clients like the gateways.
• Indexer CLI — The command line interface for managing the Indexer agent. It allows Indexers to manage cost models, manual allocations, actions queue, and indexing rules.

Indexer Infrastructure.

At the center of the indexer’s infrastructure the Graph Node which monitors Ethereum, which extracts and loads data per subgraph description and serves as a GraphQL API. The Graph Node needs to be connected to the Ethereum EVM Node endpoint, IPFS Node for sourcing data, PostgreSQL data for storing, and Indexer component which facilitates interaction with the network

Community Slide https://docs.google.com/presentation/d/1VJ09Bz4NQ6ud3XIH_0RMWrC3THqt1A-iZhoKgjo-FN8/edit#slide=id.gff22dbd644_1_34

How does the Graph Work from Indexer point of view?

1. The Indexer operator decides which subraph to index, Indexer Operator interacts with Indexer agent using Indexer CLI. The Indexer Agent has it’s own PostgreSQL Database to store data about subgraph manifests, and all subgraphs selected by Indexer Operator
2. Indexer agent instructs Graph Node (index node) to get raw data from blockchain node and index the data and store the indexed data to PostgreSQL (subgraph entity data)
3. The Graph Node (Query Node) response to queries from Indexer services from PostgreSQL (Subgraph Data) Database which was indexed.

Learn More about Indexer Architecture

Indexing - The Graph Docs

Automate Graph Deployment

GitHub Repository of Infrastructure as Code for Graph Deployment

GitHub - sivanaikk/graph-deployment

You can choose any of the following public clouds to deploy the graph infrastructure

Amazon Web Services (AWS) Terraform

Install prerequisites

• Install Terraform
• Install kubectl
• Installing the AWS CLI version 2

Apply Terraform

- Auth in AWS

Get the AWS access key using the AWS guide: AWS Account and Access Keys.

Having obtained the access key, run in the terminal:

aws configure

And input your Access Key ID and your Secret Access Key.

- Fill variables

Copy the example variables file terraform.example.tfvars to terraform.tfvars in the terraform/aws directory.

Most variables are optional and Terraform will prompt you for variables if you have not filled in the required ones.

You can also override the variables defined in variables.tf

- Run terraform apply

Go to the terraform/aws directory and run the following command in the terminal:

# you will need to run terraform init if terraform has yet to be initialiized
# a .terraform.lock.hcl file will be present if terraform is initialized
terraform init

terraform apply

After calculating the diff, you will be asked if you want to apply the changes. Type yes and press Enter.

- Get kubectl config for the created cluster

You can check your AWS CLI version by running the following command:

aws --version

aws eks --region <region-code> update-kubeconfig --name <cluster_name>

If you use the default values provided in variables.tf, run:

aws eks --region us-east-1 update-kubeconfig --name graph-indexer

- Test that kubectl config works

kubectl get ns,node

Google Cloud Platform (GCP) Terraform

Install prerequisites

• Install Terraform
• Install kubectl
• Install Cloud SDK

Apply Terraform

- Auth in GCP

Run in terminal:

gcloud auth login

This will open the Google login page in your browser. Log in to your account.

- Fill variables

Copy the sample variables file terraform.example.tfvars to terraform.tfvars in the terraform/gcp directory.

Most variables are optional and Terraform will prompt you for variables if you have not filled required ones.

Fill the gcp_project variable. You can also override other variables defined in variables.tf.

- Run terraform apply

Go to the terraform/gcp directory and run the following command in the terminal:

# you will need to run terraform init if terraform has yet to be initialiized
# a .terraform.lock.hcl file will be present if terraform is initialized
terraform init

terraform apply

After calculating the diff, you will be asked if you want to apply the changes. Type yes and press Enter.

After the command succeeds, the Kubernetes cluster will be created.

- Get kubectl config for the created cluster

For the zonal cluster, run:

gcloud container clusters get-credentials <cluster_name> --zone <cluster_zone> --project <gcp_project>

For the regional cluster, run:

gcloud container clusters get-credentials <cluster_name> --region <cluster_region> --project <gcp_project>

- Test that kubectl config works

kubectl get ns,node

Azure Terraform

Install prerequisites

• Install Terraform
• Install kubectl
• Install Azure CLI

Apply Terraform

- Auth in Azure

Run in terminal:

az login

This will open the Microsoft login page in your browser. Log in to your account.

- Fill variables

Copy the sample variables file terraform.example.tfvars to terraform.tfvars in the terraform/azure directory.

Most variables are optional and Terraform will prompt you for variables if you have not filled in the required ones.

You can also override the variables defined in variables.tf

- Run terraform apply

Go to the terraform/azure directory and run the following command in the terminal:

# you will need to run terraform init if terraform has yet to be initialiized
# a .terraform.lock.hcl file will be present if terraform is initialized
terraform init

terraform apply

After calculating the diff, you will be asked if you want to apply the changes. Type yes and press Enter.

- Get kubectl config for the created cluster

az aks get-credentials --resource-group <name> --name <name>

- Test that kubectl config works

kubectl get ns,node

Caution: Destroy Infrastructure if you follow this article just to test or if you don’t use the infrastructure.

Destroy Terraform

If you have active services or ingresses in your cluster that is associated with a load balancer, you must delete those services/ingresses before deleting the cluster so that the load balancers are deleted properly. Otherwise, you can have orphaned resources in your VPC that prevent you from being able to delete the VPC.

This step deletes all resources that were created by Terraform.

Go to the terraform/<public-cloud> directory and run the following command in the terminal:

terraform destroy

After calculating the diff, you will be asked if you want to apply the changes. Type yes and press Enter.

Learn More about Graph here :

The Graph Learn

Thank you for reading!! If you want to know more about The Graph feel free to contact me

• LinkedIn
• sivanaikk0903@gmail.com