您正在查看: 标签 同步主网数据到数据库 下的文章

搭建EOS MongoDB 分片集群

 2018-08-24　  Surou　  阅读 　  EOS-新手教程　  0 评论

由于EOS blocks数据同步到Mongo数据库后，体积很大，查询缓慢，并且机器配置比较低，同时运行nodeos写入MongoDB有点遭不住，so，部署下mongo集群，并做分片配置

1. 安装 MongoDB

准备了三台分别1T存储的云主机。三台主机在同一内网，IP分别如下

172.31.32.31 172.31.32.29 172.31.32.30
mongos mongos mongos
config server config server config server
shard server1 主节点 shard server1 副节点 shard server1 仲裁
shard server2 仲裁 shard server2 主节点 shard server2 副节点
shard server3 副节点 shard server3 仲裁 shard server3 主节点

端口分配：

占用 端口
mongos 20000
config 21000
shard1 27001
shard2 27002
shard3 27003

分别在每台机器建立conf、mongos、config、shard1、shard2、shard3六个目录，
因为mongos不存储数据，只需要建立日志文件目录即可。/mnt/data为机器挂载的1T存储盘

mkdir -p /mnt/data/mongodb/conf \
mkdir -p /mnt/data/mongodb/mongos/log \
mkdir -p /mnt/data/mongodb/config/data \
mkdir -p /mnt/data/mongodb/config/log \
mkdir -p /mnt/data/mongodb/shard1/data \
mkdir -p /mnt/data/mongodb/shard1/log \
mkdir -p /mnt/data/mongodb/shard2/data \
mkdir -p /mnt/data/mongodb/shard2/log \
mkdir -p /mnt/data/mongodb/shard3/data \
mkdir -p /mnt/data/mongodb/shard3/log

安装 mongodb-server

sudo apt install mongodb-server

配置环境变量

vi /etc/profile
# MongoDB 环境变量内容
export MONGODB_HOME=~/opt/mongodb
export PATH=$MONGODB_HOME/bin:$PATH

使立即生效

source /etc/profile

2. config server配置服务器

mongodb3.4以后要求配置服务器也创建副本集，不然集群搭建不成功。
(三台机器)添加配置文件 vi /mnt/data/mongodb/conf/config.conf

## Profile content
pidfilepath = /mnt/data/mongodb/config/log/configsrv.pid
dbpath = /mnt/data/mongodb/config/data
logpath = /mnt/data/mongodb/config/log/congigsrv.log
logappend = true

bind_ip = 0.0.0.0
port = 21000
fork = true

#declare this is a config db of a cluster;
configsvr = true

#Replica set name
replSet = configs

#Set the maximum number of connections
maxConns = 20000

启动三台服务器的config server

mongod -f /mnt/data/mongodb/conf/config.conf

登录任意一台配置服务器，初始化配置副本集

连接 MongoDB

mongo --port 21000

config 变量

config = {
    _id : "configs",
    members : [
    {_id : 0, host : "172.31.32.31:21000" },
    {_id : 1, host : "172.31.32.29:21000" },
    {_id : 2, host : "172.31.32.30:21000" }
    ]
}

初始化副本集

rs.initiate(config)

其中，"_id" : "configs"应与配置文件中配置的 replicaction.replSetName 一致，"members" 中的 "host" 为三个节点的 ip 和 port
响应内容如下

> config = {
...     _id : "configs",
...     members : [
...     {_id : 0, host : "172.31.32.31:21000" },
...     {_id : 1, host : "172.31.32.29:21000" },
...     {_id : 2, host : "172.31.32.30:21000" }
...     ]
... }
{
    "_id" : "configs",
    "members" : [
        {
            "_id" : 0,
            "host" : "172.31.32.31:21000"
        },
        {
            "_id" : 1,
            "host" : "172.31.32.29:21000"
        },
        {
            "_id" : 2,
            "host" : "172.31.32.30:21000"
        }
    ]
}
> rs.initiate(config)
{
    "ok" : 1,
    "operationTime" : Timestamp(1535116186, 1),
    "$gleStats" : {
        "lastOpTime" : Timestamp(1535116186, 1),
        "electionId" : ObjectId("000000000000000000000000")
    },
    "$clusterTime" : {
        "clusterTime" : Timestamp(1535116186, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}
configs:SECONDARY> rs.status()

此时会发现终端上的输出已经有了变化。

//从单个一个
>
//变成了
configs:SECONDARY>

查询状态

configs:SECONDARY> rs.status()

输出如下

configs:SECONDARY> rs.status()
{
    "set" : "configs",
    "date" : ISODate("2018-08-24T13:10:23.181Z"),
    "myState" : 1,
    "term" : NumberLong(1),
    "configsvr" : true,
    "heartbeatIntervalMillis" : NumberLong(2000),
    "optimes" : {
        "lastCommittedOpTime" : {
            "ts" : Timestamp(1535116209, 1),
            "t" : NumberLong(1)
        },
        "readConcernMajorityOpTime" : {
            "ts" : Timestamp(1535116209, 1),
            "t" : NumberLong(1)
        },
        "appliedOpTime" : {
            "ts" : Timestamp(1535116209, 1),
            "t" : NumberLong(1)
        },
        "durableOpTime" : {
            "ts" : Timestamp(1535116209, 1),
            "t" : NumberLong(1)
        }
    },
    "members" : [
        {
            "_id" : 0,
            "name" : "172.31.32.31:21000",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 37,
            "optime" : {
                "ts" : Timestamp(1535116209, 1),
                "t" : NumberLong(1)
            },
            "optimeDurable" : {
                "ts" : Timestamp(1535116209, 1),
                "t" : NumberLong(1)
            },
            "optimeDate" : ISODate("2018-08-24T13:10:09Z"),
            "optimeDurableDate" : ISODate("2018-08-24T13:10:09Z"),
            "lastHeartbeat" : ISODate("2018-08-24T13:10:22.372Z"),
            "lastHeartbeatRecv" : ISODate("2018-08-24T13:10:21.182Z"),
            "pingMs" : NumberLong(0),
            "syncingTo" : "172.31.32.30:21000",
            "configVersion" : 1
        },
        {
            "_id" : 1,
            "name" : "172.31.32.29:21000",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 37,
            "optime" : {
                "ts" : Timestamp(1535116209, 1),
                "t" : NumberLong(1)
            },
            "optimeDurable" : {
                "ts" : Timestamp(1535116209, 1),
                "t" : NumberLong(1)
            },
            "optimeDate" : ISODate("2018-08-24T13:10:09Z"),
            "optimeDurableDate" : ISODate("2018-08-24T13:10:09Z"),
            "lastHeartbeat" : ISODate("2018-08-24T13:10:22.372Z"),
            "lastHeartbeatRecv" : ISODate("2018-08-24T13:10:22.128Z"),
            "pingMs" : NumberLong(0),
            "syncingTo" : "172.31.32.30:21000",
            "configVersion" : 1
        },
        {
            "_id" : 2,
            "name" : "172.31.32.30:21000",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 195,
            "optime" : {
                "ts" : Timestamp(1535116209, 1),
                "t" : NumberLong(1)
            },
            "optimeDate" : ISODate("2018-08-24T13:10:09Z"),
            "infoMessage" : "could not find member to sync from",
            "electionTime" : Timestamp(1535116196, 1),
            "electionDate" : ISODate("2018-08-24T13:09:56Z"),
            "configVersion" : 1,
            "self" : true
        }
    ],
    "ok" : 1,
    "operationTime" : Timestamp(1535116209, 1),
    "$gleStats" : {
        "lastOpTime" : Timestamp(1535116186, 1),
        "electionId" : ObjectId("7fffffff0000000000000001")
    },
    "$clusterTime" : {
        "clusterTime" : Timestamp(1535116209, 1),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}
configs:PRIMARY>

3. 配置分片副本集

3.1 设置第一个分片副本集

(三台机器)设置第一个分片副本集
配置文件vi /mnt/data/mongodb/conf/shard1.conf

pidfilepath = /mnt/data/mongodb/shard1/log/shard1.pid
dbpath = /mnt/data/mongodb/shard1/data
logpath = /mnt/data/mongodb/shard1/log/shard1.log
logappend = true

bind_ip = 0.0.0.0
port = 27001
fork = true
replSet = shard1

#declare this is a shard db of a cluster;
shardsvr = true
maxConns = 20000

启动三台服务器的shard1 server

mongod -f /mnt/data/mongodb/conf/shard1.conf

登陆任意一台服务器，初始化副本集(除了172.31.32.30)
连接 MongoDB

mongo --port 27001

使用admin数据库

use admin

定义副本集配置

config = {
    _id : "shard1",
     members : [
         {_id : 0, host : "172.31.32.31:27001" },
         {_id : 1, host : "172.31.32.29:27001" },
         {_id : 2, host : "172.31.32.30:27001" , arbiterOnly: true }
     ]
 }

初始化副本集配置

rs.initiate(config)

响应内容如下

> use admin
switched to db admin
> config = {
...     _id : "shard1",
...      members : [
...          {_id : 0, host : "172.31.32.31:27001" },
...          {_id : 1, host : "172.31.32.29:27001" },
...          {_id : 2, host : "172.31.32.30:27001" , arbiterOnly: true }
...      ]
...  }
{
        "_id" : "shard1",
        "members" : [
                {
                        "_id" : 0,
                        "host" : "172.31.32.31:27001"
                },
                {
                        "_id" : 1,
                        "host" : "172.31.32.29:27001"
                },
                {
                        "_id" : 2,
                        "host" : "172.31.32.30:27001",
                        "arbiterOnly" : true
                }
        ]
}
> rs.initiate(config)
{ "ok" : 1 }

此时会发现终端上的输出已经有了变化。

//从单个一个
>
//变成了
shard1:SECONDARY>

查询状态

shard1:SECONDARY> rs.status()

响应内容如下

shard1:SECONDARY> rs.status()
{
        "set" : "shard1",
        "date" : ISODate("2018-08-24T13:24:09.626Z"),
        "myState" : 1,
        "term" : NumberLong(1),
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1535117039, 2),
                        "t" : NumberLong(1)
                },
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1535117039, 2),
                        "t" : NumberLong(1)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1535117039, 2),
                        "t" : NumberLong(1)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1535117039, 2),
                        "t" : NumberLong(1)
                }
        },
        "members" : [
                {
                        "_id" : 0,
                        "name" : "172.31.32.31:27001",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 396,
                        "optime" : {
                                "ts" : Timestamp(1535117039, 2),
                                "t" : NumberLong(1)
                        },
                        "optimeDate" : ISODate("2018-08-24T13:23:59Z"),
                        "infoMessage" : "could not find member to sync from",
                        "electionTime" : Timestamp(1535117038, 1),
                        "electionDate" : ISODate("2018-08-24T13:23:58Z"),
                        "configVersion" : 1,
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "172.31.32.29:27001",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 21,
                        "optime" : {
                                "ts" : Timestamp(1535117039, 2),
                                "t" : NumberLong(1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1535117039, 2),
                                "t" : NumberLong(1)
                        },
                        "optimeDate" : ISODate("2018-08-24T13:23:59Z"),
                        "optimeDurableDate" : ISODate("2018-08-24T13:23:59Z"),
                        "lastHeartbeat" : ISODate("2018-08-24T13:24:08.467Z"),
                        "lastHeartbeatRecv" : ISODate("2018-08-24T13:24:04.801Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "172.31.32.31:27001",
                        "configVersion" : 1
                },
                {
                        "_id" : 2,
                        "name" : "172.31.32.30:27001",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 21,
                        "lastHeartbeat" : ISODate("2018-08-24T13:24:08.467Z"),
                        "lastHeartbeatRecv" : ISODate("2018-08-24T13:24:04.781Z"),
                        "pingMs" : NumberLong(0),
                        "configVersion" : 1
                }
        ],
        "ok" : 1
}

3.2 设置第二个分片副本集

设置第二个分片副本集
配置文件 vi /mnt/data/mongodb/conf/shard2.conf

pidfilepath = /mnt/data/mongodb/shard2/log/shard2.pid
dbpath = /mnt/data/mongodb/shard2/data
logpath = /mnt/data/mongodb/shard2/log/shard2.log
logappend = true

bind_ip = 0.0.0.0
port = 27002
fork = true

replSet=shard2

#declare this is a shard db of a cluster;
shardsvr = true
maxConns=20000

启动三台服务器的shard2 server

mongod -f /mnt/data/mongodb/conf/shard2.conf

连接 MongoDB

mongo --port 27002

使用admin数据库

use admin

定义副本集配置

config = {
    _id : "shard2",
     members : [
         {_id : 0, host : "172.31.32.31:27002"  , arbiterOnly: true },
         {_id : 1, host : "172.31.32.29:27002" },
         {_id : 2, host : "172.31.32.30:27002" }
     ]
 }

初始化副本集配置

rs.initiate(config)

响应内容如下

> use admin
switched to db admin
> config = {
...     _id : "shard2",
...      members : [
...          {_id : 0, host : "172.31.32.31:27002"  , arbiterOnly: true },
...          {_id : 1, host : "172.31.32.29:27002" },
...          {_id : 2, host : "172.31.32.30:27002" }
...      ]
...  }
{
        "_id" : "shard2",
        "members" : [
                {
                        "_id" : 0,
                        "host" : "172.31.32.31:27002",
                        "arbiterOnly" : true
                },
                {
                        "_id" : 1,
                        "host" : "172.31.32.29:27002"
                },
                {
                        "_id" : 2,
                        "host" : "172.31.32.30:27002"
                }
        ]
}
> rs.initiate(config)
{ "ok" : 1 }
rs.status()

查看状态信息返回

shard2:SECONDARY> rs.status()
{
        "set" : "shard2",
        "date" : ISODate("2018-08-24T13:34:17.459Z"),
        "myState" : 1,
        "term" : NumberLong(1),
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1535117651, 1),
                        "t" : NumberLong(1)
                },
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1535117651, 1),
                        "t" : NumberLong(1)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1535117651, 1),
                        "t" : NumberLong(1)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1535117651, 1),
                        "t" : NumberLong(1)
                }
        },
        "members" : [
                {
                        "_id" : 0,
                        "name" : "172.31.32.31:27002",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 39,
                        "lastHeartbeat" : ISODate("2018-08-24T13:34:15.701Z"),
                        "lastHeartbeatRecv" : ISODate("2018-08-24T13:34:15.286Z"),
                        "pingMs" : NumberLong(0),
                        "configVersion" : 1
                },
                {
                        "_id" : 1,
                        "name" : "172.31.32.29:27002",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 153,
                        "optime" : {
                                "ts" : Timestamp(1535117651, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDate" : ISODate("2018-08-24T13:34:11Z"),
                        "infoMessage" : "could not find member to sync from",
                        "electionTime" : Timestamp(1535117629, 1),
                        "electionDate" : ISODate("2018-08-24T13:33:49Z"),
                        "configVersion" : 1,
                        "self" : true
                },
                {
                        "_id" : 2,
                        "name" : "172.31.32.30:27002",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 39,
                        "optime" : {
                                "ts" : Timestamp(1535117651, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1535117651, 1),
                                "t" : NumberLong(1)
                        },
                        "optimeDate" : ISODate("2018-08-24T13:34:11Z"),
                        "optimeDurableDate" : ISODate("2018-08-24T13:34:11Z"),
                        "lastHeartbeat" : ISODate("2018-08-24T13:34:15.701Z"),
                        "lastHeartbeatRecv" : ISODate("2018-08-24T13:34:16.331Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "172.31.32.29:27002",
                        "configVersion" : 1
                }
        ],
        "ok" : 1
}

3.3 设置第三个分片副本集

设置第三个分片副本集
配置文件 vi /mnt/data/mongodb/conf/shard3.conf

pidfilepath = /mnt/data/mongodb/shard3/log/shard3.pid
dbpath = /mnt/data/mongodb/shard3/data
logpath = /mnt/data/mongodb/shard3/log/shard3.log
logappend = true

bind_ip = 0.0.0.0
port = 27003
fork = true
replSet=shard3

#declare this is a shard db of a cluster;
shardsvr = true
maxConns=20000

启动三台服务器的shard3 server

mongod -f /mnt/data/mongodb/conf/shard3.conf

登陆任意一台服务器，初始化副本集(除了172.31.32.31)

mongo --port 27003

使用admin数据库

use admin

定义副本集配置

config = {
    _id : "shard3",
     members : [
         {_id : 0, host : "172.31.32.31:27003" },
         {_id : 1, host : "172.31.32.29:27003" , arbiterOnly: true},
         {_id : 2, host : "172.31.32.30:27003" }
     ]
 }

初始化副本集配置

rs.initiate(config)

响应内容如下

> use admin
switched to db admin
> config = {
...     _id : "shard3",
...      members : [
...          {_id : 0, host : "172.31.32.31:27003" },
...          {_id : 1, host : "172.31.32.29:27003" , arbiterOnly: true},
...          {_id : 2, host : "172.31.32.30:27003" }
...      ]
...  }
{
    "_id" : "shard3",
    "members" : [
        {
            "_id" : 0,
            "host" : "172.31.32.31:27003"
        },
        {
            "_id" : 1,
            "host" : "172.31.32.29:27003",
            "arbiterOnly" : true
        },
        {
            "_id" : 2,
            "host" : "172.31.32.30:27003"
        }
    ]
}
> rs.initiate(config)
{ "ok" : 1 }

查看状态

rs.status()

返回信息如下

shard3:SECONDARY> rs.status()
{
    "set" : "shard3",
    "date" : ISODate("2018-08-24T13:45:19.572Z"),
    "myState" : 1,
    "term" : NumberLong(1),
    "heartbeatIntervalMillis" : NumberLong(2000),
    "optimes" : {
        "lastCommittedOpTime" : {
            "ts" : Timestamp(1535118313, 1),
            "t" : NumberLong(1)
        },
        "readConcernMajorityOpTime" : {
            "ts" : Timestamp(1535118313, 1),
            "t" : NumberLong(1)
        },
        "appliedOpTime" : {
            "ts" : Timestamp(1535118313, 1),
            "t" : NumberLong(1)
        },
        "durableOpTime" : {
            "ts" : Timestamp(1535118313, 1),
            "t" : NumberLong(1)
        }
    },
    "members" : [
        {
            "_id" : 0,
            "name" : "172.31.32.31:27003",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 97,
            "optime" : {
                "ts" : Timestamp(1535118313, 1),
                "t" : NumberLong(1)
            },
            "optimeDurable" : {
                "ts" : Timestamp(1535118313, 1),
                "t" : NumberLong(1)
            },
            "optimeDate" : ISODate("2018-08-24T13:45:13Z"),
            "optimeDurableDate" : ISODate("2018-08-24T13:45:13Z"),
            "lastHeartbeat" : ISODate("2018-08-24T13:45:18.293Z"),
            "lastHeartbeatRecv" : ISODate("2018-08-24T13:45:18.675Z"),
            "pingMs" : NumberLong(0),
            "syncingTo" : "172.31.32.30:27003",
            "configVersion" : 1
        },
        {
            "_id" : 1,
            "name" : "172.31.32.29:27003",
            "health" : 1,
            "state" : 7,
            "stateStr" : "ARBITER",
            "uptime" : 97,
            "lastHeartbeat" : ISODate("2018-08-24T13:45:18.293Z"),
            "lastHeartbeatRecv" : ISODate("2018-08-24T13:45:18.602Z"),
            "pingMs" : NumberLong(0),
            "configVersion" : 1
        },
        {
            "_id" : 2,
            "name" : "172.31.32.30:27003",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 126,
            "optime" : {
                "ts" : Timestamp(1535118313, 1),
                "t" : NumberLong(1)
            },
            "optimeDate" : ISODate("2018-08-24T13:45:13Z"),
            "infoMessage" : "could not find member to sync from",
            "electionTime" : Timestamp(1535118232, 1),
            "electionDate" : ISODate("2018-08-24T13:43:52Z"),
            "configVersion" : 1,
            "self" : true
        }
    ],
    "ok" : 1
}

3.4 配置路由服务器 mongos

（三台机器）先启动配置服务器和分片服务器,后启动路由实例启动路由实例:
修改配置文件vi /mnt/data/mongodb/conf/mongos.conf

pidfilepath = /mnt/data/mongodb/mongos/log/mongos.pid
logpath = /mnt/data/mongodb/mongos/log/mongos.log
logappend = true

bind_ip = 0.0.0.0
port = 20000
fork = true

#The configuration server that listens can only have 1 or 3 configs to configure the replica set name of the server.
configdb = configs/172.31.32.31:21000,172.31.32.29:21000,172.31.32.30:21000

maxConns = 20000

启动三台服务器的mongos server

mongos -f /mnt/data/mongodb/conf/mongos.conf

4. 串联路由服务器

目前搭建了mongodb配置服务器、路由服务器，各个分片服务器，不过应用程序连接到mongos路由服务器并不能使用分片机制，还需要在程序里设置分片配置，让分片生效。
登陆任意一台mongos

mongo --port 20000

使用admin数据库

use admin

串联路由服务器与分配副本集

sh.addShard("shard1/172.31.32.31:27001,172.31.32.29:27001,172.31.32.30:27001");
sh.addShard("shard2/172.31.32.31:27002,172.31.32.29:27002,172.31.32.30:27002");
sh.addShard("shard3/172.31.32.31:27003,172.31.32.29:27003,172.31.32.30:27003");

查看集群状态

sh.status()

响应内容如下

mongos> sh.addShard("shard1/172.31.32.31:27001,172.31.32.29:27001,172.31.32.30:27001");
{
    "shardAdded" : "shard1",
    "ok" : 1,
    "$clusterTime" : {
        "clusterTime" : Timestamp(1535118917, 6),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    },
    "operationTime" : Timestamp(1535118917, 6)
}
mongos> sh.addShard("shard2/172.31.32.31:27002,172.31.32.29:27002,172.31.32.30:27002");
{
    "shardAdded" : "shard2",
    "ok" : 1,
    "$clusterTime" : {
        "clusterTime" : Timestamp(1535118917, 10),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    },
    "operationTime" : Timestamp(1535118917, 10)
}
mongos> sh.addShard("shard3/172.31.32.31:27003,172.31.32.29:27003,172.31.32.30:27003");
{
    "shardAdded" : "shard3",
    "ok" : 1,
    "$clusterTime" : {
        "clusterTime" : Timestamp(1535118918, 5),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    },
    "operationTime" : Timestamp(1535118918, 5)
}
mongos> sh.status()
--- Sharding Status --- 
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("5b8003a67629bf9778831c77")
  }
  shards:
        {  "_id" : "shard1",  "host" : "shard1/172.31.32.29:27001,172.31.32.31:27001",  "state" : 1 }
        {  "_id" : "shard2",  "host" : "shard2/172.31.32.29:27002,172.31.32.30:27002",  "state" : 1 }
        {  "_id" : "shard3",  "host" : "shard3/172.31.32.30:27003,172.31.32.31:27003",  "state" : 1 }
  active mongoses:
        "3.6.3" : 3
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours: 
                No recent migrations
  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }

5. 启用集合分片生效

目前配置服务、路由服务、分片服务、副本集服务都已经串联起来了，但我们的目的是希望插入数据，数据能够自动分片。连接在mongos上，准备让指定的数据库、指定的集合分片生效。
登陆任意一台mongos

mongo --port 20000

使用admin数据库

use admin

指定eos数据库分片生效

db.runCommand( { enablesharding :"eos"});

指定数据库里需要分片的集合和片键，哈希id 分片

db.runCommand( { shardcollection : "eos.blocks",key : {"_id": "hashed"} } );

6. 修改eos config.ini

三台机器其中一台，运行nodeos并同步blocks数据到mongodb

plugin = eosio::mongo_db_plugin
mongodb-uri = mongodb://localhost:20000/eos

重置数据

nodeos --data-dir /mnt/data/data --hard-replay-blockchain --mongodb-wipe

需要清空数据的话加下 --mongodb-wipe
然后查看任意一台mongo数据库

mongo --port 20000
mongos> show databases;
admin   0.000GB
config  0.001GB
eos     0.071GB

已经正确分片存储

参考：segmentfault

配置EOS同步主网数据到Mysql

 2018-08-18　  Surou　  阅读 　  EOS-新手教程　  0 评论

EOS编译安装，请参考编译EOS主网EOS-Mainnet代码并支持sql_db_plugin

一. 修改Config配置

先运行下nodeos，将会自动创建~/.local/share/eosio/nodeos/config目录和config.ini文件。
修改config.ini中如下内容

//添加 （2018-8-18 此时可用，如果有想分享的节点，请私聊我）
p2p-peer-address = fullnode.eoslaomao.com:443
p2p-peer-address = mars.fnp2p.eosbixin.com:443
//修改 可忽略
agent-name = "BcSkill"
//如果需要返回错误信息,修改
verbose-http-errors = true
//添加插件支持
plugin = eosio::chain_plugin
plugin = eosio::net_plugin
plugin = eosio::sql_db_plugin
//修改mongodb插件相关配置
sql_db-uri = mysql://db=eos user=root host=127.0.0.1 password='pwd'

二. 安装配置并启动MySql

1. 安装MySql

先安装MySql 参考 Ubuntu 安装MySql 8.0

2. 配置MySql

  • 接下来我们安装 soci。soci 是 C++ 连接 MySQL 的 Library。Ubuntu 可以快速安装，命令如下：
    sudo apt-get -y install libsoci-dev
  • 安装 mysql-client
    sudo apt-get -y install mysql-client
sudo apt-get install libmysqlclient-dev

由于链上数据较大，比如1000W块左右的数据，离线压缩包大约14GB，同步到MySql大概需要200GB左右。所以需要将MySql数据单独磁盘存储。
修改MySql数据存储位置,/mnt/data为我这边挂载的存储盘
新建目录mkdir /mnt/data/MySql
并参考 Ubuntu16.04下修改MySQL数据的默认存储位置

配置Eos数据库

  • 进入mysql
    mysql -p
  • 创建数据库
    CREATE DATABASE eos DEFAULT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
  • 查看数据库
    mysql > show databases;

三. 下载主网离线数据包

参考 EOS 主网数据更新，离线数据包
因为使用离线包可以加快追上主网的进度，如果从零自己同步到主网进度，难以想象要多久...
假设已经将离线数据下载并解压到了 /mnt/data/data目录下。

四. 启动nodeos开始同步数据

nodeos --data-dir /mnt/data/data --hard-replay-blockchain --replay-blockchain
第一次运行 必须添加 --replay-blockchain参数，不然不会创建数据库表，参考下面代码

如果需要清空mysql数据库的话，添加--sql_db-block-start=0 --replay-blockchain
相关代码如下eos\plugins\sql_db_plugin\sql_db_plugin.cpp

void sql_db_plugin::plugin_initialize(const variables_map& options)
{
    ilog("initialize");
    try {
        std::string uri_str = options.at(SQL_DB_URI_OPTION).as<std::string>();
        if (uri_str.empty())
        {
            wlog("db URI not specified => eosio::sql_db_plugin disabled.");
            return;
        }
        ilog("connecting to ${u}", ("u", uri_str));
        uint32_t block_num_start = options.at(BLOCK_START_OPTION).as<uint32_t>();
        auto db = std::make_unique<database>(uri_str, block_num_start);

        if (options.at(HARD_REPLAY_OPTION).as<bool>() ||
                options.at(REPLAY_OPTION).as<bool>() ||
                options.at(RESYNC_OPTION).as<bool>() ||
                !db->is_started())
        {
            if (block_num_start == 0) {
                ilog("Resync requested: wiping database");
                if( options.at( RESYNC_OPTION ).as<bool>() ||
                        options.at( REPLAY_OPTION ).as<bool>()) {
                    ilog( "Resync requested: wiping database" );
                    db->wipe();
                }
            }
        }

此时已开始，同步中

五. 服务器建议配置

内存：32GB
存储：1T+

编译EOS主网EOS-Mainnet代码并支持sql_db_plugin

 2018-08-18　  Surou　  阅读 　  EOS-新手教程　  0 评论

涉及的代码仓库

EOSIO官方将不再收录sql_db_plugin，第三方自己维护（github），v1.2.0已删除sql_db_plugin，官方只维护mongo_db_plugin。

合并代码，提交到自己的仓库

基于EOS-Mainnet对应的分支合并NebulaProtocol中sql_db_plugin相关代码。
由于我熟悉Windows TortoiseGit工具，所以我在Windows上准备好代码，提交到github自己的仓库，（也方便自己维护），再在Ubuntu上拉取分支代码。

步骤尽量的简单，方便新手按部就班，老鸟绕飞~

github 上新建仓库https://github.com/cppfuns/Pure-EOS.git

新建目录Pure-EOS,获取EOS-Mainnet代码

git clone https://github.com/EOS-Mainnet/eos.git

添加NebulaProtocol仓库

进入Pure-EOS目录，右键TortoiseGit->Settings，添加NebulaProtocol仓库

获取所有仓库更新信息到本地

再次在Pure-EOS目录，右键TortoiseGit->fetch,获取所有仓库更新信息到本地

切换分支

切换EOS-Mainnet当前最新的release分支，当前为mainnet-1.1.6
再次在Pure-EOS目录，右键TortoiseGit->Switch/Checkout...，选择mainnet-1.1.6分支

合并代码


再次在Pure-EOS目录，右键TortoiseGit->Show log,找到NebulaProtocol/sql_plugin分支

右键Merge to mainnet-1.1.6

点击确定后，如果有冲突，自行解决，

提交代码

此时已经将NebulaProtocol中sql_db_plugin分支代码，合并到了EOS-Mainnet的最新分支mainnet-1.1.6上了。提交本地的mainnet-1.1.6分支到github 上新建仓库https://github.com/cppfuns/Pure-EOS.git。

获取代码

在Ubuntu上直接获取此仓库

git clone https://github.com/cppfuns/Pure-EOS.git

切换分支

cd eos源码目录
git checkout mainnet-1.1.6

开始编译代码

./eosio_build.sh -s EOS

执行安装

./eosio_install.sh

检测sql_db_plugin

编译完成后,执行

nodeos --help | grep "sql_db-uri"

如果输出–sql_db-uri arg Sql DB URI connection string If not内容，证明编译的EOS已支持sql_db_plugin

配置EOS同步主网数据到mongoDB

 2018-08-18　  Surou　  阅读 　  EOS-新手教程　  2 评论

EOS编译安装，请参考 编译EOS主网EOS-Mainnet代码

一. 修改Config配置

先运行下nodeos，将会自动创建~/.local/share/eosio/nodeos/config目录和config.ini文件。
修改config.ini中如下内容

//添加 （2018-8-18 此时可用，如果有想分享的节点，请私聊我）
p2p-peer-address = fullnode.eoslaomao.com:443
p2p-peer-address = mars.fnp2p.eosbixin.com:443
//修改 可忽略
agent-name = "BcSkill"
//如果需要返回错误信息,修改
verbose-http-errors = true
//添加插件支持
plugin = eosio::chain_plugin
plugin = eosio::net_plugin
//修改mongodb插件相关配置
plugin = eosio::mongo_db_plugin
mongodb-uri = mongodb://127.0.0.1:27017/EOS
mongodb-filter-on = *
#mongodb-filter-out = spammer::
mongodb-filter-out = eosio:onblock:
mongodb-filter-out = gu2tembqgage::
mongodb-filter-out = blocktwitter::
mongodb-queue-size = 2048
abi-serializer-max-time-ms = 5000
mongodb-block-start = 1
mongodb-store-block-states = false
mongodb-store-blocks = false
mongodb-store-transactions = false
mongodb-store-transaction-traces = true
mongodb-store-action-traces = true

read-mode = read-only

参考：MongoDB Filtering and Optimizations
参考：issues/5797

二. 安装配置并启动MongoDB

1. 安装MongoDB

先安装MongoDB 参考（Ubuntu 安装 Mongodb 3+）

2. 配置MongoDB

进入到MongoDB bin 目录，可以添加到环境变量，方便操作。

cd ~/opt/mongodb/bin

由于链上数据较大，比如1000W块左右的数据，离线压缩包大约14GB，同步到MongoDB大概需要200GB左右。所以需要将MongoDB数据单独磁盘存储。
修改MongoDB数据存储位置,/mnt/data为我这边挂载的存储盘
新建目录mkdir /mnt/data/mongo/db

3. 启动MongoDB

mongod --dbpath /mnt/data/mongo/db

这时MongoDB服务会默认监听27017端口

三. 下载主网离线数据包

参考 EOS 主网数据更新，离线数据包
因为使用离线包可以加快追上主网的进度，如果从零自己同步到主网进度，难以想象要多久...
假设已经将离线数据下载并解压到了 /mnt/data/data目录下。

四. 启动nodeos开始同步数据

nodeos --data-dir /mnt/data/data --hard-replay-blockchain

如果需要清空mongo数据库的话，添加--mongodb-wipe
相关代码如下eos\plugins\mongo_db_plugin\mongo_db_plugin.cpp

void mongo_db_plugin::plugin_initialize(const variables_map& options)
{
   try {
      if( options.count( "mongodb-uri" )) {
         ilog( "initializing mongo_db_plugin" );
         my->configured = true;

         if( options.at( "replay-blockchain" ).as<bool>() || options.at( "hard-replay-blockchain" ).as<bool>() || options.at( "delete-all-blocks" ).as<bool>() ) {
            if( options.at( "mongodb-wipe" ).as<bool>()) {
               ilog( "Wiping mongo database on startup" );
               my->wipe_database_on_startup = true;
            } else if( options.count( "mongodb-block-start" ) == 0 ) {
               EOS_ASSERT( false, chain::plugin_config_exception, "--mongodb-wipe required with --replay-blockchain, --hard-replay-blockchain, or --delete-all-blocks"
                                 " --mongodb-wipe will remove all EOS collections from mongodb." );
            }
         }

此时已开始，同步中

五. 服务器建议配置

内存：32GB
存储：1T+

编译EOS主网EOS-Mainnet代码

 2018-08-18　  Surou　  阅读 　  EOS-新手教程　  0 评论

下载主网EOS代码

所谓的主网代码就是EOS-Mainnet维护的EOS分支仓库，相对EOSIO的仓库分支区别就是，EOS-Mainnet对发布的release分支做了进一步测试和审核，测试好后，BP们才部署。也就是基本上BP们都是部署的这个仓库分支，所以如果是同步主网，建议选此仓库分支。避免一些不必要的bug和数据不兼容等意外。

1. 获取代码
git clone https://github.com/EOS-Mainnet/eos.git

如果提示git没有安装，先安装gitapt install git

2. 切换对应的分支

由于主仓库分支较多，切换到当前仓库的最新的分支或者tag
当前最新为（mainnet-1.1.6）

cd eos源码目录
git checkout mainnet-1.1.6
3. 开始编译代码
./eosio_build.sh -s EOS

执行脚本后，脚本会自动安装所需的开发环境工具

4. 执行安装

编译完成后，

执行安装

./eosio_install.sh

最新文章
跨链桥跨链过程中数据变化polygon cdk aggregator 报execution reverted (0x09bde339)polygon cdk Aggregator使用SequencerPrivateKey的时机polygon da 判断DA服务是否正常polygon cdk sequence sender 发送 sequenceBatchesValidium的时机polygon cdk 查看最新提交L1batch numCDK 部署或重置后第一笔交易error: latest Synced BlockNumber (7089137) is higher than the latest Proposed block (1008) in the networkpolygon cdk 跨链桥交易处理流程分析因RPC限速导致sequence-sender异常退出

最新回复
fzd: 请问这个解决了吗 StarkWare explained: layer 2 solution provider of dYdX and iMMUTABLE R11; BitKeep News: [...]Layer 2: https://... 一文读懂 StarkWare：dYdX 和 Immutable 背后的 L2 方案 R11; BitKeep 博客: [...]Layer 2:Comparing Laye... http://andere.strikingly.com/: Regards, Great stuff! surou: 需要先执行提案合约申请，等待出块节点地址同意后，才会进... heco: WARN [11-19|11:26:09.459] N... P: 你好，我在heco链上遇到了“tx fee excee... Peng: 楼主安装成功了吗？我正在同步区块链，一天了，差不多才同... joyhu: 你好，请问下安装好之后如何获取到bee.yaml配置文... kaka: 支票最终怎么提币呢？

标签

归档
November 2024October 2024September 2024August 2024July 2024June 2024May 2024April 2024March 2024January 2024December 2023November 2023October 2023September 2023August 2023July 2023June 2023April 2023March 2023February 2023January 2023December 2022November 2022October 2022August 2022July 2022June 2022May 2022March 2022February 2022January 2022December 2021November 2021October 2021September 2021August 2021July 2021June 2021May 2021April 2021March 2021February 2021January 2021December 2020November 2020October 2020September 2020July 2020June 2020May 2020April 2020March 2020February 2020January 2020December 2019November 2019October 2019September 2019August 2019July 2019June 2019May 2019April 2019March 2019February 2019January 2019December 2018November 2018October 2018September 2018August 2018July 2018June 2018

其他
文章 RSS 评论 RSS