Categories
Docker Neo4j

Neo4j Cluster(apoc+gds) Docker with Portainer

Like most of the RDBMS and NoSQL Databases, Neo4j also provides Clustering. Clustering provides three main features –

High Availability – Always available even if there are node failures.

Horizontal Scalability – Read Only Replicas distribute loads isolated from write nodes.

Consistency – when enabled, the client application call is guaranteed to read at least its own successful writes.

causal clustering

More documentation for Neo4j Casual Cluster can be found at Neo4j Official documentation.

For demo purpose, we will create a 3-Node Neo4j Cluster with apoc and graph data science latest plugin using Docker along with Portainer.

Portainer is an open-source and lightweight management UI which allows to easily manage Docker environments.

Github link -> docker-compose.yml

version: "3.8"

services:
  core1:
    hostname: core1
    image: neo4j:enterprise
    networks:
      - neo4j_cluster_ntx
    container_name: core1
    volumes:
      - ./core1/neo4j/data:/var/lib/neo4j/data
      - ./core1/neo4j/import:/var/lib/neo4j/import
    environment:
      - NEO4J_ACCEPT_LICENSE_AGREEMENT=yes
      - NEO4JLABS_PLUGINS=["apoc","graph-data-science"]
      - NEO4J_dbms_default__listen__address=0.0.0.0
      - NEO4J_dbms_memory_pagecache_size=1G
      - NEO4J_dbms.memory.heap.initial_size=2G
      - NEO4J_dbms_memory_heap_max__size=4G
      - NEO4J_dbms_directories_import=/var/lib/neo4j/import
      - NEO4J_dbms_security_procedures_unrestricted=gds.*,apoc.*
      - NEO4J_dbms_security_procedures_allowlist=gds.*,apoc.*
      - NEO4J_dbms_mode=CORE
      - NEO4J_causal__clustering_minimum__core__cluster__size__at__formation=3
      - NEO4J_causal__clustering_minimum__core__cluster__size__at__runtime=3
      - NEO4J_causal__clustering_discovery__advertised__address=core1:5000
      - NEO4J_causal__clustering_transaction__advertised__address=core1:6000
      - NEO4J_causal__clustering_raft__advertised__address=core1:7000
      - NEO4J_causal__clustering_initial__discovery__members=core1:5000,core2:5000,core3:5000
      - NEO4J_causal__clustering_disable__middleware__logging=false
    ports:
      - 7474:7474
      - 6477:6477
      - 7687:7687
  core2:
    hostname: core2
    image: neo4j:enterprise
    networks:
      - neo4j_cluster_ntx
    container_name: core2
    volumes:
      - ./core2/neo4j/data:/var/lib/neo4j/data
      - ./core2/neo4j/import:/var/lib/neo4j/import
    environment:
      - NEO4J_ACCEPT_LICENSE_AGREEMENT=yes
      - NEO4JLABS_PLUGINS=["apoc","graph-data-science"]
      - NEO4J_dbms_default__listen__address=0.0.0.0
      - NEO4J_dbms_memory_pagecache_size=1G
      - NEO4J_dbms.memory.heap.initial_size=2G
      - NEO4J_dbms_memory_heap_max__size=4G
      - NEO4J_dbms_directories_import=/var/lib/neo4j/import
      - NEO4J_dbms_security_procedures_unrestricted=gds.*,apoc.*
      - NEO4J_dbms_security_procedures_allowlist=gds.*,apoc.*
      - NEO4J_dbms_mode=CORE
      - NEO4J_causal__clustering_minimum__core__cluster__size__at__formation=3
      - NEO4J_causal__clustering_minimum__core__cluster__size__at__runtime=3
      - NEO4J_causal__clustering_discovery__advertised__address=core2:5000
      - NEO4J_causal__clustering_transaction__advertised__address=core2:6000
      - NEO4J_causal__clustering_raft__advertised__address=core2:7000
      - NEO4J_causal__clustering_initial__discovery__members=core1:5000,core2:5000,core3:5000
      - NEO4J_causal__clustering_disable__middleware__logging=false
      - NEO4J_dbms_connector_http_listen__address=:7475
      - NEO4J_dbms_connector_https_listen__address=:6478
      - NEO4J_dbms_connector_bolt_listen__address=:7688
    ports:
      - 7475:7475
      - 6478:6478
      - 7688:7688
  core3:
    hostname: core3
    image: neo4j:enterprise
    networks:
      - neo4j_cluster_ntx
    container_name: core3
    volumes:
      - ./core3/neo4j/data:/var/lib/neo4j/data
      - ./core3/neo4j/import:/var/lib/neo4j/import
    environment:
      - NEO4J_ACCEPT_LICENSE_AGREEMENT=yes
      - NEO4JLABS_PLUGINS=["apoc","graph-data-science"]
      - NEO4J_dbms_default__listen__address=0.0.0.0
      - NEO4J_dbms_memory_pagecache_size=1G
      - NEO4J_dbms.memory.heap.initial_size=2G
      - NEO4J_dbms_memory_heap_max__size=4G
      - NEO4J_dbms_directories_import=/var/lib/neo4j/import
      - NEO4J_dbms_security_procedures_unrestricted=gds.*,apoc.*
      - NEO4J_dbms_security_procedures_allowlist=gds.*,apoc.*
      - NEO4J_dbms_mode=CORE
      - NEO4J_causal__clustering_minimum__core__cluster__size__at__formation=3
      - NEO4J_causal__clustering_minimum__core__cluster__size__at__runtime=3
      - NEO4J_causal__clustering_discovery__advertised__address=core3:5000
      - NEO4J_causal__clustering_transaction__advertised__address=core3:6000
      - NEO4J_causal__clustering_raft__advertised__address=core3:7000
      - NEO4J_causal__clustering_initial__discovery__members=core1:5000,core2:5000,core3:5000
      - NEO4J_causal__clustering_disable__middleware__logging=false
      - NEO4J_dbms_connector_http_listen__address=:7476
      - NEO4J_dbms_connector_https_listen__address=:6479
      - NEO4J_dbms_connector_bolt_listen__address=:7689
    ports:
      - 7476:7476
      - 6479:6479
      - 7689:7689
  portainer:
    image: portainer/portainer
    networks:
      - neo4j_cluster_ntx
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./pt-data:/data
    ports:
      - "9000:9000"
networks:
  neo4j_cluster_ntx:

docker-compose up takes some time to completely setup and bootstrap the data on all neo4j nodes.

Note During the build process, at the portainer step, you have to login to localhost:9000 to create the admin user within 5 mins.

Neo4j Cluster

Portainer Setup

Portainer setup –

Create a user, and in the next screen, select “Local” as we are using an open-source portainer setup, and click “Connect”.

once the setup is finished, the portainer dashboard is displayed.

Selecting the containers from the left menu, shows various containers

Portainer has a great feature for performance metrics. It can display CPU, Memory and Network usage with refresh rate start from 1sec to 1 min

Neo4j Cluster

After few mins of bootstrapping and syncing with the 3 nodes, the neo4j cluster is ready to use. (wait for Remote interface available)

 core1        | 2021-01-18 00:37:01.407+0000 INFO  Called db.clearQueryCaches(): Query cache already empty.
 core2        | 2021-01-18 00:37:04.411+0000 INFO  Started downloading snapshot for database 'neo4j'…
 core3        | 2021-01-18 00:37:04.415+0000 INFO  Started downloading snapshot for database 'neo4j'…
 core2        | 2021-01-18 00:37:08.827+0000 INFO  Download of snapshot for database 'neo4j' complete.
 core3        | 2021-01-18 00:37:08.912+0000 INFO  Download of snapshot for database 'neo4j' complete.
 core3        | 2021-01-18 00:37:12.648+0000 INFO  Called db.clearQueryCaches(): Query cache already empty.
 core2        | 2021-01-18 00:37:13.007+0000 INFO  Called db.clearQueryCaches(): Query cache already empty.
 core3        | 2021-01-18 00:37:16.529+0000 INFO  Connected to core2/172.21.0.4:7000 [raft version:3.0]
 core2        | 2021-01-18 00:37:16.611+0000 INFO  Connected to core3/172.21.0.5:7000 [raft version:3.0]
 core1        | 2021-01-18 00:37:20.275+0000 INFO  Sending metrics to CSV file at /var/lib/neo4j/metrics
 core1        | 2021-01-18 00:37:20.349+0000 INFO  Bolt enabled on 0.0.0.0:7687.
 core1        | 2021-01-18 00:37:22.063+0000 INFO  Remote interface available at http://localhost:7474/
 core1        | 2021-01-18 00:37:22.064+0000 INFO  Started.
 core2        | 2021-01-18 00:37:31.467+0000 INFO  Sending metrics to CSV file at /var/lib/neo4j/metrics
 core3        | 2021-01-18 00:37:31.485+0000 INFO  Sending metrics to CSV file at /var/lib/neo4j/metrics
 core2        | 2021-01-18 00:37:31.553+0000 INFO  Bolt enabled on 0.0.0.0:7688.
 core3        | 2021-01-18 00:37:31.561+0000 INFO  Bolt enabled on 0.0.0.0:7689.
 core2        | 2021-01-18 00:37:33.468+0000 INFO  Remote interface available at http://localhost:7475/
 core2        | 2021-01-18 00:37:33.469+0000 INFO  Started.
 core3        | 2021-01-18 00:37:33.471+0000 INFO  Remote interface available at http://localhost:7476/
 core3        | 2021-01-18 00:37:33.472+0000 INFO  Started.

Open a web-browser and navigate to localhost:7474 to access the core1 node (default username and password is “neo4j”). After setting the neo4j password, execute call dbms.cluster.overview()

╒══════════════════════════════════════╤═════════════════════════════════════════════════╤════════════════════════════════════════╤════════╕
│"id"                                  │"addresses"                                      │"databases"                             │"groups"│
╞══════════════════════════════════════╪═════════════════════════════════════════════════╪════════════════════════════════════════╪════════╡
│"a861c553-4cd5-4f8b-baae-3279fa92c65a"│["bolt://localhost:7688","http://localhost:7475"]│{"neo4j":"FOLLOWER","system":"FOLLOWER"}│[]      │
├──────────────────────────────────────┼─────────────────────────────────────────────────┼────────────────────────────────────────┼────────┤
│"e38564bb-1f42-4738-9629-29e5cf1f0bb3"│["bolt://localhost:7689","http://localhost:7476"]│{"neo4j":"LEADER","system":"FOLLOWER"}  │[]      │
├──────────────────────────────────────┼─────────────────────────────────────────────────┼────────────────────────────────────────┼────────┤
│"2614662c-7f3a-4024-8900-f3555e26630d"│["bolt://localhost:7687","http://localhost:7474"]│{"neo4j":"FOLLOWER","system":"LEADER"}  │[]      │
└──────────────────────────────────────┴─────────────────────────────────────────────────┴────────────────────────────────────────┴────────┘

 167 total views,  1 views today