In this article, you will learn the utilities to perform system benchmark testing on on-premise installation, to ensure the provided core server and spark worker servers meets our requirement.
Utility needed for the test: heathchecktoolkit.zip (attached)
Reference Setup and server terminology:
server1: Paxata Core Server, Pipeline Server, MongoDB
worker1: Spark Worker OR YARN Node Manager.
worker2: Spark Worker OR YARN Node Manager.
1. Pipeline Executor Cache Disk Random Write Performance Test
a. On worker1 and worker2, Find the fio rpm in the zip attached and install it.
yum localinstall fio-2.0.13-1.el6.x86_64.rpm
b. On worker1 server, run the following command to test iops performance on "/tmp/test". Change /tmp/test to your actual directory for storing pipeline executor cache.
sudo fio --directory=/tmp/test --name fio_test_file --direct=1 --rw=randwrite --bs=16k --size=1G --numjobs=16 --time_based --runtime=180 --group_reporting --norandommap
Expected Result: depends on number of worker cores, Disk random write iops should be from 12,000 for 8 worker cores to 48,000 for 32 worker cores.
2. Network Bandwidth Performance Test
a. On worker1, worker2 and server1. Find the iperf rpm in the zip attached and install it.
yum localinstall iperf3-3.0.11-1.el6.x86_64.rpm
1st test
===============================
Assuming worker1 running on 10.0.2.176, starting a server on port 58921:
sudo iperf3 -s -p 58921
worker2 connecting to worker1 on port 58921:
sudo iperf3 -c 10.0.2.176 -i 1 -t 60 -V -p 58921
2nd test
===============================
Assuming worker1 running on 10.0.2.176, starting a server on port 58921:
sudo iperf3 -s -p 58921
server1 connecting to worker1 on port 58921:
sudo iperf3 -c 10.0.2.176 -p 58921
Expected Result: depends on number of worker cores, network bandwidth should be from 4Gbit/sec for 8 worker cores to 10Gbit/sec for 32 worker cores.
3. Core Server and Spark Worker CPU Core Performance Test
a. On worker1, worker2 and server1. Find the taskset rpms in the zip attached and install it.
yum localinstall util-linux-ng-2.17.2-12.28.el6.x86_64.rpm;
yum localinstall libuuid-2.17.2-12.28.el6.x86_64.rpm;
yum localinstall libblkid-2.17.2-12.28.el6.x86_64.rpm;
b. Check Number of Physical CPU Cores reported by Operating System
egrep -e "core id" -e ^physical /proc/cpuinfo|xargs -l2 echo|sort -u
c. Run Gzip tests on adjacent and non-adjacent cores
adjacent cores test: taskset -pc 0,1 $$; for i in {1..2}; do dd if=/dev/zero bs=1M count=2070 2> >(grep bytes >&2 ) | gzip -c > /dev/null & done;
non-adjacent cores test: taskset -pc 0,2 $$; for i in {1..2}; do dd if=/dev/zero bs=1M count=2070 2> >(grep bytes >&2 ) | gzip -c > /dev/null & done;
Sample Output
=====================================================
2170552320 bytes (2.2 GB) copied, 15.0161 s, 145 MB/s
2170552320 bytes (2.2 GB) copied, 15.2538 s, 142 MB/s
Expected Result:
For adjacent core test, Gzip test should reach above 60MB/s
For non-adjacent core test, Gzip test should reach above 120MB/s
Note: As a part of application monitoring health check, please use system resource monitoring tool. You can use your in-house standard resource monitoring tool or you can use something like Netdata
https://github.com/firehol/netdata/wiki/Why-netdata%3F .
Please note Netdata is not certified with Paxata, its generic system resource monitoring tool.
日本語
