Table of Contents

Benchmark Testing Environment 3

Overview of Testing Plan. 3

Benchmark Testing Contents. 4

Functional Test 4

Meta-Data Operations. 4

Node-Edge Queries. 4

Node-Edge Insertion. 5

Path Queries. 6

K-hop Neighbor Queries. 8

Graph Algorithms. 10

Degrees. 10

Similarities. 11

Communities and Label Propagations. 12

Functionality & Feature Testing and Comparison. 12

Overall User Experience & Visualization. 13

Usability & Maintainability. 13

Extensibility. 14

Summaries and Findings. 15

Benchmark Testing Environment

• OS:  RedHat 7.0
• Hardware/Cloud Setup:

• CPU: 8-core (Intel Xeon Gold 6148 2.4GHz, cache 28160KB)
• DRAM: 64GB
• Disks: 1TB SSD
• Networks： ACS Network (powered by Microsoft Azure Cloud Services）
• Cluster Setup: 3-instances
  • Neo4j: Causal cluster
  • Ultipa: HTAP cluster

• Software:
• Neo4j 4.07 Enterprise Edition
• Ultipa Server v2.1
• Ultipa KG v2.1
• Docker 19.0+
• Datasets:
• Alimama published dataset
• User transactional behavior dataset (5.15M vertices, 100M edges)
• Vertex Entities: users and products.
• Edge Relationships: Page Views, Add-to-cart, Add-to-favorite, buy (purchase), etc.
• Average node degree ~= 20 (100M/5M)

Overview of Testing Plan

The testing includes side-by-side feature comparison and performance benchmarking between Neo4j v4.07 Enterprise edition and Ultipa v2.1 edition.

Based on retail banking needs, the testing will be primarily focusing on graph queries and graph computing performances:

• Meta-data related operations: CRUD operations on nodes and edges;
• Graph operations: Path, k-hop queries
• Graph algorithms: Benchmark against a few typical graph algorithms

For non-performance functionality related features, the following areas are tested:

• Support of visualization (GUI features)
• Support of high-availability and cluster deployment and management
• Support of extensibility
• Support of toolkits/toolchains
• Overall usability and user experience.

Both systems are deployed in cluster setup:

• Neo4j Enterprise v4.07 with 3 instances causal cluster setup
• Ultipa v2.1 with 3 instances HTAP cluster setup

Note that, the bank’s on-premise cloud environment is supposedly ACS (Advanced Cloud Services) which is based on Microsoft Azure with some special network adaptations, think ACS for the bank’s private cloud as AWS for the public cloud. Due to Neo4j’s incompatibility with ACS, we had to resolve to use a bare-metal cluster setup to conduct the following testing.

Benchmark Testing Contents

Functional Test

Meta-Data Operations

Node-Edge Queries

Randomized queries against nodes and edges by their IDs. The queries are repeated 1,000 times for each type, total time and average individual query time are counted. If TPS (transactions per second) is counted, it’s also recorded.

Note that all numbers below are in milliseconds (ms), unless otherwise annotated.

Neo4j testing results:

Ultipa testing results:

Node-Edge Insertion

Randomly inserting 1,000 nodes and 1,000 edges. Count the total time and average time per insertion. All operations latencies are counted in milliseconds (ms).

Neo4j testing results:

Ultipa testing results:

Path Queries

Randomly query against N pairs of nodes, the search depths are 1-hop, 3-hop, or 5-hop.

When searching with filters, use the PV attribute tagged on each edge (relationship).

For 1 and 3-hop path queries, random ID pairs are used for testing.

For 5-hop queries, due to this testing dataset’s topology characteristics, to ensure results are available for easier latency comparison, the following pairs of IDs are pre-selected:

• u602926,p111765
• p111765,u728516
• p3276990,u857323
• u965769,p2724538
• u602926,p4755938
• p1944109,u532914
• u900245,p242356
• u88211,p1364561 (hotspot to hotspot)

To understand the testing dataset’s “challenge”, let’s use the following simple math, based on average node degree, which stands at 20:

• 1-hop: average edges to filter or traverse: 20
• 2-hop: average edges to filter or traverse: 400
• 3-hop: average edges to filter or traverse: 8,000
• 4-hop: average edges to filter or traverse: 160,000
• 5-hop: average edges to filter or traverse: 3,200,000

Given hotspot nodes, they are usually orders of magnitudes more heavily connected, therefore making traversing or querying against such nodes way more challenging. It’s highly probable to see that when counting any node’s neighbors that are 4 or 5-hop away, hotspot nodes will be encountered! Due to this challenge we, therefore, limit the 5-hop deep path query to be only 1, and latency to be within 1800 seconds, for Neo4j to make sure we do NOT have to wait extensively long for results. The same types of 5-hop deep path queries were conducted 7 times by Ultipa to see how consistent its performance was.

Neo4j testing results

Note that for path queries, a 30-min cap is set for each query, if the query does NOT return within 30-min, the query will be killed. Given Neo4j enough time, say, a few hours, it may eventually return with results.

Ultipa testing results:

K-hop Neighbor Queries

The K-hop testing is to calculate the number of neighbors of a certain starting node that are exactly K-hop away, given certain filtering rules.

Note the definition of K-hop is that if a node appears to be the starting node’s k-hop neighbor, their shortest path in-between must be of K hops in length.

The filtering condition is by edge’s PV attribute. All queries are randomized and repeated 1,000 or 100 times, given the query depth.

Neo4j testing results:

Ultipa testing results:

Graph Algorithms

Algorithms testing are conducted around the following areas:

• Basic graph algorithms, such as degrees
• Similarities, such as Jaccard
• Communities, such as Louvain and label propagations

Degrees

In-degree and out-degree or degrees are commonly used basic graph algorithms, the testing is conducted to measure Neo4j and Ultipa’s performance.

Neo4j testing results:

Ultipa testing results:

Similarities

Jaccard Similarities are used in the following benchmarks.

There are 2 types of Jaccard Similarities:

• Node-to-Node Similarity
• Top-10 Most Similar Nodes (to a certain node)

Neo4j testing results:

Ultipa testing results:

Communities and Label Propagations

LPA (Label Propagation Algorithm) and Louvain are two commonly used graph algorithms, though Louvain is considerably more sophisticated than LPA, the benchmark results are captured in the following table:

Functionality & Feature Testing and Comparison

The following key features are compared:

• Usability & Maintainability
• Visualization & User-Experience
• Extensibility

Overall User Experience & Visualization

Usability & Maintainability

Extensibility

Summaries and Findings

To recap:

• Performance-wise: Neo4j, in general, is 10 times slower than Ultipa;
• When search depth is beyond 3-hop, the performance gap is enlarged;
• When the query complexity is elevated, the performance gap is also widened.
• For graph algorithms, Ultipa shows a similar performance edge over Neo4j.

Pros & Cons:

• Neo4j community is much bigger, and it's primarily OLAP oriented.
• Ultipa is relatively new to the market, and it's HTAP (OLTP+OLAP) focused.
• For performance/TP scenarios, Ultipa should be considered with priority.
• For simple and small datasets, there is no significant difference between Neo4j and Ultipa.
• For large and sophisticated datasets and data modeling, Ultipa has a clear advantage.
• Ultipa also gains clear advantages over Neo4j from a visualization perspective.