nGrinder对监控机器收集自定义数据及源码分析

转载：https://blog.****.net/neven7/article/details/50782451

0.背景

性能测试工具nGrinder支持在无需修改源码的情况下，对目标服务器收集自定义数据，最多支持5类；

在性能测试详细报告页，目标服务器->你的机器ip便签页下，默认只收集CPU, Memory, Received Byte/s, Sent Byte Per Secode/s等4类数据；

可能你还需要监控其它的性能统计数据，用于分析(比如load, Full Gc)；本文先介绍实现方法；再分析nGrinder源码，看它是怎么实现的。

1.实现

1-1. 安装monitor

在你的nGrinder系统下，下载监控

安装到你测试服务所在的机器，解压tar包，执行sh run_monitor_bg.sh;

其实脚本是启了个java服务，以monitor模式启动；

之前介绍过Agent有2种模式：

gent mode: 运行进程和线程，压测目标服务；

monitor mode: 监控目标系统性能(cpu/memory)。

[root@10 ngrinder-monitor]# cat run_monitor.sh
#!/bin/sh
curpath=`dirname $0`
cd ${curpath}
java -server -cp "lib/*" org.ngrinder.NGrinderAgentStarter --mode monitor --command run $@

• 1
• 2
• 3
• 4
• 5
• 6

Agent的home路径为/root/.ngrinder_agent，你在执行sh run_monitor_bg.sh默认获取的配置信息为/root/.ngrinder_agent/agent.conf； 如果加上-o，sh run_monitor_bg.sh 读取你安装monitor目录下的__agent.conf， 该配置文件定义了Agent的模式，ip, 端口。

[root@10 .ngrinder_agent]# cat agent.conf
common.start_mode=monitor
#If you want to monitor bind to the different local ip not automatically selected ip. Specify below field.
#monitor.binding_port=hostname_or_ip
monitor.binding_port=13243

• 1
• 2
• 3
• 4
• 5
• 6

自定义数据需放在/root/.ngrinder_agent/monitor/custom.data文件里，格式如下：

类型1数据,类型2数据,类型3数据,类型4数据,类型5数据

• 1

最多支持5类，每类数据用“,”分隔，注意的是： 数据是实时的写文件，不是累积数据到文件中(类似shell中的>, 不是>>)，即同一时刻，只有一行数据。

1-2. 定制收集脚本

以收集load和full GC为例：

[root@10 bin]# cat updateCustomData.sh
#!/bin/sh
#@author hugang

customDataRoot=/root/.ngrinder_agent/monitor/custom.data;
#  获取load信息
load=`/bin/cat /proc/loadavg | awk '{print $1}'`;
#  获取full gc count
if [[ $1 -gt 0  ]]; then
  fgc=`jstat -gcutil $1 | tail -1 | awk '{print $8}'`;
  echo $load,$fgc > $customDataRoot;
else
  echo $load > $customDataRoot;
fi;

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15

开始性能测试时，每秒去执行该脚本，收集数据到custom.data中：

 watch -n 1 sh updateCustomData.sh 5528

• 1

5528为需监控java服务进程pid;

当你性能测试结束后，monitor收集的数据会放到/root/.ngrinder/perftest/0_999/{test_id}/report/monitor_system_{test_id}/report/monitor_system_{ip}.data文件中:

[root@10 report]# cat monitor_system_10.13.1.139.data
ip,system,collectTime,freeMemory,totalMemory,cpuUsedPercentage,receivedPerSec,sentPerSec,customValues
10.13.1.139,LINUX,20160302151441,97102768,132112072,26.895683,32954,27897,4.93,49
10.13.1.139,LINUX,20160302151443,97075896,132112072,30.513468,45702,32306,4.93,49
10.13.1.139,LINUX,20160302151445,97034772,132112072,30.411074,110306,65391,5.02,49
10.13.1.139,LINUX,20160302151447,96972504,132112072,22.073017,84813,57503,5.02,49
...

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8

1-3.结果展示

2.源码分析

nGrinder使用Sigar工具（https://support.hyperic.com/display/SIGAR/Home）收集系统信息，该工具可以收集以下数据：

System memory, swap, cpu, load average, uptime, logins
Per-process memory, cpu, credential info, state, arguments, environment, open files
File system detection and metrics
Network interface detection, configuration info and metrics
TCP and UDP connection tables
Network route table

• 1
• 2
• 3
• 4
• 5
• 6
• 7

sigar工具（http://download.****.net/download/neven7/9450930）示例：

[root@10 testsigar]# ls
libsigar-amd64-linux.so  sigar-1.6.4.jar  sigar-1.6.4.jar.zip
[root@10 testsigar]#
[root@10 testsigar]# java  -jar ./sigar-1.6.4.jar
sigar> free
             total       used       free
Mem:     132112072   96855372   35256700
-/+ buffers/cache:   34855500   97256572
Swap:      8388600     264980    8123620
RAM:      129016MB
sigar> 

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11

收集系统数据的java文件为： 
ngrinder-core/src/main/java/org/ngrinder/monitor/collector/SystemDataCollector.java

继承和实现关系：

SystemDataCollector extends DataCollector 

DataCollector implements Runnable

• 1
• 2
• 3
• 4

SystemDataCollector的线程执行体：

public void run() {
        // 初始化sigar
        initSigar();
        SystemMonitoringData systemMonitoringData = (SystemMonitoringData) getMXBean(SYSTEM);
        // execute()通过sigar api获取系统信息
        systemMonitoringData.setSystemInfo(execute());
    }

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8

execute()获取系统信息SystemInfo(System info object to save date collected by monitor)：

/**
     * Execute the collector to get the system info model.
     *
     * @return SystemInfo in current time
     */
    public synchronized SystemInfo execute() {
        SystemInfo systemInfo = new SystemInfo();
        systemInfo.setCollectTime(System.currentTimeMillis());
        try {
            BandWidth networkUsage = getNetworkUsage();
            BandWidth bandWidth = networkUsage.adjust(prev.getBandWidth());
            systemInfo.setBandWidth(bandWidth);
            systemInfo.setCPUUsedPercentage((float) sigar.getCpuPerc().getCombined() * 100);
            Cpu cpu = sigar.getCpu();
            systemInfo.setTotalCpuValue(cpu.getTotal());
            systemInfo.setIdleCpuValue(cpu.getIdle());
            Mem mem = sigar.getMem();
            systemInfo.setTotalMemory(mem.getTotal() / 1024L);
            systemInfo.setFreeMemory(mem.getActualFree() / 1024L);
            systemInfo.setSystem(OperatingSystem.IS_WIN32 ? SystemInfo.System.WINDOW : SystemInfo.System.LINUX);
            systemInfo.setCustomValues(getCustomMonitorData());
        } catch (Throwable e) {
            LOGGER.error("Error while getting system perf data:{}", e.getMessage());
            LOGGER.debug("Error trace is ", e);
        }
        prev = systemInfo;
        return systemInfo;
    }

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29

其中：getCustomMonitorData()获取自定义数据，读取custom.data文件中一行数据

private String getCustomMonitorData() {
        if (customDataFile != null && customDataFile.exists()) {
            BufferedReader customDataFileReader = null;
            try {
                customDataFileReader = new BufferedReader(new FileReader(customDataFile));
                return customDataFileReader.readLine(); // these data will be parsed at
                // monitor client side.
            } catch (IOException e) {
                // Error here is very natural
                LOGGER.debug("Error to read custom monitor data", e);
            } finally {
                IOUtils.closeQuietly(customDataFileReader);
            }
        }
        return prev.getCustomValues();
    }

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17

综上：类SystemDataCollector作用就是作为线程执行体，线程每次执行通过sigar获取系统信息：SystemInfo，赋值给SystemMonitoringData成员变量SystemInfo。

前面介绍启动monitor时，其实是执行了org.ngrinder.NGrinderAgentStarter类，我们再分析下该文件，ngrinder-core/src/main/java/org/ngrinder/NGrinderAgentStarter.java

/**
     * Agent starter.
     *
     * @param args arguments
     */
    public static void main(String[] args) {
        NGrinderAgentStarter starter = new NGrinderAgentStarter();
        final NGrinderAgentStarterParam param = new NGrinderAgentStarterParam();
        checkJavaVersion();
        JCommander commander = new JCommander(param);
        commander.setProgramName("ngrinder-agent");
        commander.setAcceptUnknownOptions(true);
        try {
            commander.parse(args);
        } catch (Exception e) {
            LOG.error(e.getMessage());
            return;
        }
        final List<String> unknownOptions = commander.getUnknownOptions();
        modeParam = param.getModeParam();
        modeParam.parse(unknownOptions.toArray(new String[unknownOptions.size()]));

        if (modeParam.version != null) {
            System.out.println("nGrinder v" + getStaticVersion());
            return;
        }

        if (modeParam.help != null) {
            modeParam.usage();
            return;
        }

        System.getProperties().putAll(modeParam.params);
        starter.init();

        final String startMode = modeParam.name();
        if ("stop".equalsIgnoreCase(param.command)) {
            starter.stopProcess(startMode);
            System.out.println("Stop the " + startMode);
            return;
        }
        starter.checkDuplicatedRun(startMode);
        if (startMode.equalsIgnoreCase("agent")) {
            starter.startAgent();
        } else if (startMode.equalsIgnoreCase("monitor")) {
            starter.startMonitor();
        } else {
            staticPrintHelpAndExit("Invalid agent.conf, '--mode' must be set as 'monitor' or 'agent'.");
        }
    }

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
• 32
• 33
• 34
• 35
• 36
• 37
• 38
• 39
• 40
• 41
• 42
• 43
• 44
• 45
• 46
• 47
• 48
• 49
• 50
• 51

monitor模式执行该方法：starter.startMonitor()

     /**
     * Start the performance monitor.
     */
    public void startMonitor() {
        printLog("***************************************************");
        printLog("* Start nGrinder Monitor... ");
        printLog("***************************************************");
        try {
            MonitorServer.getInstance().init(agentConfig);
            MonitorServer.getInstance().start();
        } catch (Exception e) {
            LOG.error("ERROR: {}", e.getMessage());
            printHelpAndExit("Error while starting Monitor", e);
        }
    }

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17

MonitorServer.getInstance().start():

    /**
     * Start monitoring.
     *
     * @throws IOException exception
     */
    public void start() throws IOException {
        if (!isRunning()) {
            jmxServer.start();
            DataCollectManager.getInstance().init(agentConfig);
            DataCollectManager.getInstance().start();
            isRunning = true;
        }
    }

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13

DataCollectManager.getInstance().start();

    /**
     * start a scheduler for the data collector jobs.
     */
    public void start() {
        int collectorCount = MXBeanStorage.getInstance().getSize();
        scheduler = Executors.newScheduledThreadPool(collectorCount);
        if (!isRunning()) {
            Collection<MXBean> mxBeans = MXBeanStorage.getInstance().getMXBeans();
            for (MXBean mxBean : mxBeans) {
                DataCollector collector = mxBean.gainDataCollector(agentConfig.getHome().getDirectory());
                scheduler.scheduleWithFixedDelay(collector, 0L, getInterval(), TimeUnit.SECONDS);
                LOG.info("{} started.", collector.getClass().getSimpleName());
            }
            LOG.info("Collection interval : {}s).", getInterval());
            isRunning = true;
        }
    }

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17

scheduler.scheduleWithFixedDelay(collector, 0L, getInterval(), TimeUnit.SECONDS);

线程池周期地执行SystemDataCollector中run()去获取系统数据。

    @Override
    public void run() {
        initSigar();
        SystemMonitoringData systemMonitoringData = (SystemMonitoringData) getMXBean(SYSTEM);
        systemMonitoringData.setSystemInfo(execute());
    }

• 1
• 2
• 3
• 4
• 5
• 6

3.总结：

后台启动的monitor， 运行的是一个java服务:

java -server -cp lib/* org.ngrinder.NGrinderAgentStarter --mode monitor --command run

• 1

通过线程池周期获取系统性信息(sigar工具获取)，存放在SystemInfo;

ngrinder-controller/src/main/java/org/ngrinder/perftest/service/samplinglistener/MonitorCollectorPlugin.java中startSampling():

@Override
    public void startSampling(final ISingleConsole singleConsole, PerfTest perfTest,
                              IPerfTestService perfTestService) {
        final List<String> targetHostIP = perfTest.getTargetHostIP();
        final Integer samplingInterval = perfTest.getSamplingInterval();
        for (final String target : targetHostIP) {
            scheduledTaskService.runAsync(new Runnable() {
                @Override
                public void run() {
                    LOGGER.info("Start JVM monitoring for IP:{}", target);
                    MonitorClientService client = new MonitorClientService(target, MonitorCollectorPlugin.this.port);
                    client.init();
                    if (client.isConnected()) {
                        File testReportDir = singleConsole.getReportPath();
                        File dataFile = null;
                        try {
                            dataFile = new File(testReportDir, MONITOR_FILE_PREFIX + target + ".data");
                            FileWriter fileWriter = new FileWriter(dataFile, false);
                            BufferedWriter bw = new BufferedWriter(fileWriter);
                            // write header info
                            bw.write(SystemInfo.HEADER);
                            bw.newLine();
                            bw.flush();
                            clientMap.put(client, bw);
                        } catch (IOException e) {
                            LOGGER.error("Error to write to file:{}, Error:{}", dataFile.getPath(), e.getMessage());
                        }
                    }
                }
            });
        }
        assignScheduledTask(samplingInterval);
    }

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
• 32
• 33
• 34
• 35

根据SystemInfo写到/root/.ngrinder/perftest/0_999/{test_id}/report/monitor_system_{test_id}/report/monitor_system_{ip}.data文件中；

ngrinder-controller/src/main/java/org/ngrinder/perftest/PerfTestService.java中getMonitorGraph()根据/root/.ngrinder/perftest/0_999/{test_id}/report/monitor_system_{test_id}/report/monitor_system_{ip}.data获取系统信息数据

     /**
     * Get system monitor data and wrap the data as a string value like "[22,11,12,34,....]", which can be used directly
     * in JS as a vector.
     *
     * @param testId       test id
     * @param targetIP     ip address of the monitor target
     * @param dataInterval interval value to get data. Interval value "2" means, get one record for every "2" records.
     * @return return the data in map
     */
    public Map<String, String> getMonitorGraph(long testId, String targetIP, int dataInterval) {
        Map<String, String> returnMap = Maps.newHashMap();
        File monitorDataFile = new File(config.getHome().getPerfTestReportDirectory(String.valueOf(testId)),
                MONITOR_FILE_PREFIX + targetIP + ".data");
        BufferedReader br = null;
        try {

            StringBuilder sbUsedMem = new StringBuilder("[");
            StringBuilder sbCPUUsed = new StringBuilder("[");
            StringBuilder sbNetReceived = new StringBuilder("[");
            StringBuilder sbNetSent = new StringBuilder("[");
            StringBuilder customData1 = new StringBuilder("[");
            StringBuilder customData2 = new StringBuilder("[");
            StringBuilder customData3 = new StringBuilder("[");
            StringBuilder customData4 = new StringBuilder("[");
            StringBuilder customData5 = new StringBuilder("[");

            br = new BufferedReader(new FileReader(monitorDataFile));
            br.readLine(); // skip the header.
            // "ip,system,collectTime,freeMemory,totalMemory,cpuUsedPercentage,receivedPerSec,sentPerSec"
            String line = br.readLine();
            int skipCount = dataInterval;
            // to be compatible with previous version, check the length before
            // adding
            while (StringUtils.isNotBlank(line)) {
                if (skipCount < dataInterval) {
                    skipCount++;
                } else {
                    skipCount = 1;
                    String[] datalist = StringUtils.split(line, ",");
                    if ("null".equals(datalist[4]) || "undefined".equals(datalist[4])) {
                        sbUsedMem.append("null").append(",");
                    } else {
                        sbUsedMem.append(Long.valueOf(datalist[4]) - Long.valueOf(datalist[3])).append(",");
                    }
                    addCustomData(sbCPUUsed, 5, datalist);
                    addCustomData(sbNetReceived, 6, datalist);
                    addCustomData(sbNetSent, 7, datalist);
                    addCustomData(customData1, 8, datalist);
                    addCustomData(customData2, 9, datalist);
                    addCustomData(customData3, 10, datalist);
                    addCustomData(customData4, 11, datalist);
                    addCustomData(customData5, 12, datalist);
                    line = br.readLine();
                }
            }
            completeCustomData(returnMap, "cpu", sbCPUUsed);
            completeCustomData(returnMap, "memory", sbUsedMem);
            completeCustomData(returnMap, "received", sbNetReceived);
            completeCustomData(returnMap, "sent", sbNetSent);
            completeCustomData(returnMap, "customData1", customData1);
            completeCustomData(returnMap, "customData2", customData2);
            completeCustomData(returnMap, "customData3", customData3);
            completeCustomData(returnMap, "customData4", customData4);
            completeCustomData(returnMap, "customData5", customData5);
        } catch (IOException e) {
            LOGGER.info("Error while getting monitor {} data file at {}", targetIP, monitorDataFile);
        } finally {
            IOUtils.closeQuietly(br);
        }
        return returnMap;
    }

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
• 32
• 33
• 34
• 35
• 36
• 37
• 38
• 39
• 40
• 41
• 42
• 43
• 44
• 45
• 46
• 47
• 48
• 49
• 50
• 51
• 52
• 53
• 54
• 55
• 56
• 57
• 58
• 59
• 60
• 61
• 62
• 63
• 64
• 65
• 66
• 67
• 68
• 69
• 70
• 71
• 72

数据提供给Controller端： 
ngrinder-controller/src/man/java/org/ngrinder/perftest/controller/PerfTestController.java

    private Map<String, String> getMonitorGraphData(long id, String targetIP, int imgWidth) {
        int interval = perfTestService.getMonitorGraphInterval(id, targetIP, imgWidth);
        Map<String, String> sysMonitorMap = perfTestService.getMonitorGraph(id, targetIP, interval);
        PerfTest perfTest = perfTestService.getOne(id);
        sysMonitorMap.put("interval", String.valueOf(interval * (perfTest != null ? perfTest.getSamplingInterval() : 1)));
        return sysMonitorMap;
    }

    /**
     * Get the monitor data of the target having the given IP.
     *
     * @param id       test Id
     * @param targetIP targetIP
     * @param imgWidth image width
     * @return json message
     */
    @RestAPI
    @RequestMapping("/api/{id}/monitor")
    public HttpEntity<String> getMonitorGraph(@PathVariable("id") long id,
                                              @RequestParam("targetIP") String targetIP, @RequestParam int imgWidth) {
        return toJsonHttpEntity(getMonitorGraphData(id, targetIP, imgWidth));
    }