Getting started with sharding in MongoDB

Setting up sharding on a MongoDB collection isn't as complicated as it sounds. In fact, it usually is an overkill. However, once you decide to take the plunge, here's a basic primer on how to go about it. Will be adding the caveats at a later point in time.

Getting started with sharding in MongoDB
Photo by Rubaitul Azad / Unsplash

Setting up sharding on a MongoDB collection isn't as complicated as it sounds. In fact, it usually is an overkill. However, once you decide to take the plunge, here's a basic primer on how to go about it. Will be adding the caveats at a later point in time.  

Create database

direct: mongos] test>  use departments;
switched to db departments

Enable sharding on the database

[direct: mongos] departments> sh.enableSharding('departments')
{
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1633085931, i: 3 }),
    signature: {
      hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
      keyId: Long("0")
    }
  },
  operationTime: Timestamp({ t: 1633085931, i: 2 })
}

Create collection in the current database

[direct: mongos] departments> db.createCollection('tech')
{ ok: 1 }

Create index, hashed on single column

[direct: mongos] departments> db.tech.createIndex({_id: "hashed"})
_id_hashed

Now, shard the collection

[direct: mongos] departments> sh.shardCollection('departments.tech', {_id: "hashed"})
{
  collectionsharded: 'departments.tech',
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1633086030, i: 32 }),
    signature: {
      hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
      keyId: Long("0")
    }
  },
  operationTime: Timestamp({ t: 1633086030, i: 28 })
}

Insert data, 10-15 documents

[direct: mongos] departments> db.tech.insertOne({name: 'shazazzzm2', empID: 2898})
{
  acknowledged: true,
  insertedId: ObjectId("6156ea70d336dd9789acaec6")
}
[direct: mongos] departments> db.adminCommand( { flushRouterConfig: "departments.tech" } );
{
  flushed: true,
  ok: 1,
  '$clusterTime': {
    clusterTime: Timestamp({ t: 1633087540, i: 1 }),
    signature: {
      hash: Binary(Buffer.from("0000000000000000000000000000000000000000", "hex"), 0),
      keyId: Long("0")
    }
  },
  operationTime: Timestamp({ t: 1633087540, i: 1 })
}


mongos> db.tech.getShardDistribution();
Shard ShardReplSet2 at ShardReplSet2/ip-172-31-22-236:27018
{
  data: '0B',
  docs: 0,
  chunks: 2,
  'estimated data per chunk': '0B',
  'estimated docs per chunk': 0
}
---
Shard ShardReplSet at ShardReplSet/ip-172-31-27-182:27018
{
  data: '0B',
  docs: 0,
  chunks: 2,
  'estimated data per chunk': '0B',
  'estimated docs per chunk': 0
}
---
Totals
{
  data: '0B',
  docs: 0,
  chunks: 4,
  'Shard ShardReplSet2': [ '0 % data', '0 % docs in cluster', '0B avg obj size on shard' ],
  'Shard ShardReplSet': [ '0 % data', '0 % docs in cluster', '0B avg obj size on shard' ]
}

Replica sets

Remember: Run rs.initiate() only on the primary node.

Check replication status

rs.printSecondaryReplicationInfo()

Command quickref

# Used to determine which shard the queried data was found on
db.users.find({userId:123}).explain().queryPlanner.winningPlan

# Check if replica set member is primary/secondary
db.hello()

Log Rotation

Configure logging in the daemon conf

sudo nano /etc/mongod.conf

....

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log
  logRotate: reopen
....  

Set up the logrotate script. (Note: On mongos, change 'pidof mongod' to 'pidof mongos', and 'create 640 mongodb mongodb' to 'create 640 root root', for some reason this was set to run as root, may need looking at)


sudo nano /etc/logrotate.d/mongod.conf

/var/log/mongodb/mongod.log {
  daily
  size 100M
  rotate 5
  missingok
  #compress
  #delaycompress
  #notifempty
  create 640 mongodb mongodb
  sharedscripts
  postrotate
    /bin/kill -SIGUSR1 `pidof mongod` >/dev/null 2>&1
  endscript
}

Test the rotation

sudo logrotate -f /etc/logrotate.d/mongod.conf

ubuntu@ip-172-31-44-33:~$ ls -lh /var/log/mongodb/
total 48M
-rw-r----- 1 mongodb mongodb  44M Jan 10 10:18 mongod.log
-rw------- 1 mongodb mongodb 4.0M Jan 10 08:18 mongod.log.1

Log rotation can also be triggered from within the shell, mongosh, by running

 db.adminCommand( { logRotate : 1 } )

Setting up replication on the MySQL database.
Base AMI Setup filesystem Create filesystem on nvme volume Mount the nvme volume Get UUID, setup file table sudo lsblk -o +UUID Test auto mount Test mount point sudo chown -R mysql:mysql /data/mysql/sudo chown -R 750 /data/mysql/ Move MySQL data dir Mount point is /data, sudo