Note: This article is about features in a version of Weathermap that is not yet released. It is aimed at those who are testing the new version, prior to it's release. Feel free to help with that testing process, but 0.96 is not a finished or fully tested release yet.
There have been quite a few changes to the RRDtool datasource plugin for 0.96 - some of them are related specifically to the Cacti integration, and poller_output support, and others are more general. I'll cover the general ones here, and the Cacti-related stuff in another article about Cacti-related new features.
First of all, there are some convenience features, especially if you don't use Cacti. These SET commands are map-global, so they should appear in the top section of you map config file, before any nodes or links:
- You can define a default 'base directory' where the RRD DS will look for .rrd files, if there isn't a full path in the TARGET. This makes your config files much easier to read.
SET rrd_default_path /usr/local/mrtg/var
- You can also change the default 'in' and 'out' DS names from traffic_in and traffic_out to something more useful for your environment. For MRTG, this would be
SET rrd_default_in_ds DS0 SET rrd_default_out_ds DS1
and for Cricket, ds0/ds1.
Next up is a completely different mode for reading RRD files. Normally, weathermap fetches the last 12 minutes' data from each RRD file, and then picks the newest non-NaN line to use as the 'current' data. Of course, RRDtool's big strength is that it keeps historical data too. You have been able define rrd_start and rrd_period for quite a while now to get a historical map using RRD data - although the results 'soften' as RRDtool aggregates older data into averages over larger periods of time. It still presents data from a single point in time though.
In 0.96, you can also get data which is the average, minimum, maximum etc over a given period. So you can make a weekly maximum, or a monthly 95th percentile map. This is done with very few changes to the map config, so you can also do this without having a second config file - much less maintenance to do!
To use the aggregate functions, you need to define a few map-global SET variables. You can do this either in the map config, or using the --define command-line parameter if you use the command-line tool, or from within the Cacti plugin interface, if you use that. The last two mean you can use the same map config file for an instantaneous map, and also for 'reporting' maps.
The SET variables are:
SET rrd_start now-1month SET rrd_period 2592000 SET rrd_aggregate_function AVERAGE
The first two define which period in time to collect data from, while the last defines what to do with it. Valid aggregation functions currently include: MINIMUM, MAXIMUM, AVERAGE and PERCENT (PERCENT also requires a percentage, like '95,PERCENT'). This feature uses the 'PRINT' rrdtool graph command and VDEF feature, so as new functions are added there, they will work here. rrd_period is a timespan in seconds (30 days here) - this will likely change to allow more friendly period descriptions.
Add another cron job, and you can easily get weekly or monthly reports without managing another map.
August 31, 2009 - 7:04pm
Hi,
Have found a problem with v0.96a when using the rrd_aggregate_function feature.
In my case with '95,PERCENT'.
RRDTool v1.2.19
When using, for example,
SET rrd_start now-8hours
SET rrd_period 28800
SET rrd_aggregate_function 95,PERCENT
wmrrd_read_from_real_rrdtool_aggregate fails to parse the output from rrdtool which is in the format:
'OUT 302.533333'
'IN 321.737022'
By adding preg match for the apostroph at the start and end of the output, all is ok:
foreach ($lines as $line)
{
if(preg_match('/^\'(IN|OUT)\s(\-?\d+[\.,]?\d*e?[+-]?\d*:?)\'$/i', $line, $matches))
{
debug("MATCHED: ".$matches[1]." ".$matches[2]."\n");
if($matches[1]=='IN') $data[IN] = floatval($matches[2]);
if($matches[1]=='OUT') $data[OUT] = floatval($matches[2]);
$data_ok = TRUE;
}
}
Sample debug output:
WEATHERMAP: Poller[0] WARNING: [Map 2] MyConfig: ReadData: LINK MyLink, target: /usr/local/share/cacti/rra/My-Router_traffic_in_40.rrd on config line 194 had no valid data, according to WeatherMapDataSource_rrd
WEATHERMAP: Poller[0] DEBUG: [ReadData@WeatherMapDataSource_rrd.php:645] [Map 2] MyConfig: RRD ReadData: Returning (NULL,NULL,0)
WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_real_rrdtool_aggregate@WeatherMapDataSource_rrd.php:353] [Map 2] MyConfig: RRD ReadDataFromRealRRDAggregate: Returning (NULL,NULL,0)
WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_real_rrdtool_aggregate@WeatherMapDataSource_rrd.php:320] [Map 2] MyConfig: >
WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_real_rrdtool_aggregate@WeatherMapDataSource_rrd.php:320] [Map 2] MyConfig: > 'OUT 302.533333'
WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_real_rrdtool_aggregate@WeatherMapDataSource_rrd.php:320] [Map 2] MyConfig: > 'IN 321.737022'
WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_real_rrdtool_aggregate@WeatherMapDataSource_rrd.php:304] [Map 2] MyConfig: RRD ReadData: Running: /usr/local/bin/rrdtool graph /dev/null -f '' --start now-8h --end start+28800 DEF:in=/usr/local/share/cacti/rra/My-Router_traffic_in_40.rrd:traffic_in:AVERAGE VDEF:agg_in=in,95,PERCENT "PRINT:agg_in:'IN %lf'" DEF:out=/usr/local/share/cacti/rra/My-Router_traffic_in_40.rrd:traffic_out:AVERAGE VDEF:agg_out=out,95,PERCENT "PRINT:agg_out:'OUT %lf'"
WEATHERMAP: Poller[0] DEBUG: [wmrrd_read_from_real_rrdtool_aggregate@WeatherMapDataSource_rrd.php:248] [Map 2] MyConfig: RRD ReadData: VDEF style, for LINK MyLink
September 1, 2009 - 11:35am
This should already be fixed in the svn version of WeatherMapDataSource_rrd.php
September 1, 2009 - 6:19pm
Cool.
I pulled my copy in April.
I'll try the latest and greatest. I'm sure you came up with a more elegant way of fixing it.
Thanks.
September 4, 2009 - 6:22pm
No, it was pretty identically elegant ;-)
December 17, 2008 - 4:27pm
Love the new aggregate functionality and being able to generate multiple maps from a single conf file is great (you can tell there's a request for more features coming can't you!) it would be even better if you could specify text for the title and time string as command line parameters. I thought I could get round this with SET and --define but couldn't make it work - perhaps I'm missing something.
Really enjoying using weathermap
regards
Steve
December 31, 2008 - 2:20pm
You can use defines to change the title. I do this in my maps...
TITLE My Network{map:titleextra}
then --define titleextra as " - Monthly." or just ".", depending on the map, for example.
You can't SET a variable to empty before 0.96test2 though (I just added that), and I also changed the order that things are evaluated so that you can override a SET that's also in the config file with a command-line --define. Perhaps that's what you are hitting?
So what you should be able to do soon is:
SET titleextra
TITLE My Network {map:titleextra}
--define titleextra " - Monthly"
and the --define will override the SET.
February 24, 2009 - 10:27am
I've been trying to use the new features to make the title dynamic. I've got the following in weathermap.conf
SET titleextra
TITLE NGFL Network non-terminal nodes {map:titleextra}
If I then run weathermap with:
./weathermap --define titleextra "some extra text"
I get the following errors:
PHP Notice: Undefined offset: 1 in /usr/local/weathermap/weathermap/weathermap on line 139
Notice: Undefined offset: 1 in /usr/local/weathermap/weathermap/weathermap on line 139
PHP Notice: Undefined offset: 2 in /usr/local/weathermap/weathermap/weathermap on line 140
Notice: Undefined offset: 2 in /usr/local/weathermap/weathermap/weathermap on line 140
Any idea what I'm doing wrong?
cheers
February 25, 2009 - 12:00am
I led you astray there... it should be --define titleextra=" - something extra" (with an equals sign).
I'l see if I can make it a bit more obvious from the error message. That's not too clear, is it?
October 2, 2008 - 10:53am
I have a good map of current traffic. Running with
the following options:
--define rrd_start=-1h --define rrd_period=3600 --define rrd_aggregate_function=AVERAGE
however produces this error message:
ERROR: Cannot parse DS in 'DEF:in=/data00/huvipuisto/rrd/6002/6002-poller-10:1.rrd:INOCTETS:AVERAGE'
/usr/pwn/weathermap/plenware.conf: ReadData: LINK core1b-core2b, target: scale:0.125:6002/6002-poller-10:1.rrd on config line 157 had no valid data, according to WeatherMapDataSource_rrd
I get the error for some data sources but not all. Some links are drawn correctly on the map, and the others just have 0 data.
What is the problem here? All the rrd files have the same kind of data.
October 3, 2008 - 6:32pm
Do your rrd filenames have colons in them? Could you paste one of the 'bad' TARGET lines?
October 8, 2008 - 1:07pm
Yes, the rrd filenames do have colons in them.
TARGET scale:0.125:6001/6001-poller-10:4.rrd
October 13, 2008 - 6:08pm
Then that is why.
It sees the part after the : as a DS name, which doesn't exist, and causes your error.
October 14, 2008 - 8:15am
Well, how can I escape the colons in the name?
October 15, 2008 - 8:17am
http://oss.oetiker.ch/rrdtool/doc/rrdgraph_data.en.html#DEF doesn't mention anything. How do you do it with whatever tool generates the rrd files?
October 15, 2008 - 10:37am
http://oss.oetiker.ch/rrdtool/doc/rrdgraph_data.en.html#DEF mentions escaping colons with backslashes in parameters, probably this would do for filenames also:
"DEF:ds0weekly=router.rrd:ds0:AVERAGE:start=11\:00:end=start+1h"
October 15, 2008 - 10:35am
I use perl system('/usr/bin/rrdtool', 'create', $rrdfile, other params here...);
October 15, 2008 - 11:33am
As far as I know, there isn't an equivalent function in PHP. I would rather use that style too, it would save a bunch of quoting/escaping issues. PHP is very patchy in this kind of thing.
In this particular case, it's not the shell you need to escape the : from though, it's RRDtool itself. How do you draw graphs, for example, where you have a DEF/CDEF? Those would have the same : problem. Weathermap's aggregation just uses VDEF and the rrd graph command.
October 16, 2008 - 7:36am
Well, we have our own graph generator, we don't use "rrdtool graph", just perl RRDs::fetch(), so this is probably why it works for us.
October 16, 2008 - 8:13am
OK, my first suggestion would be to try putting the \: in the TARGET in weathermap, it just passes the string straight through to rrdtool. That may well break the non-aggregated map if you are using the same config for two maps, of course...
If it works, I'll look at adding in the escaping in the RRD datasource plugin.
August 19, 2008 - 10:09pm
I have a problem with the following command line:
/usr/bin/php5 weathermap --config configs/mymap.cfg --define rrd_period=120 --define rrd_start=1212962400 --define rrd_aggregate_function= MAXIMUM --output output/mymap_1212962400.png
the last two options get cut. So the result ist just a mymap.png (set as IMAGEOUTPUTFILE in mymap.cfg) in the weathermap folder with the AVERAGES and not the MAX values.
the same with debug:
/srv/www/cacti/plugins/weathermap # /usr/bin/php5 weathermap --debug --config configs/mymap.cfg --define rrd_period=120 --define rrd_start=1212962400 --define rrd_aggregate_function= MAXIMUM --output output/mymap_1212962400.png
DEBUG: [@weathermap:141] >> rrd_period = '120'
DEBUG: [@weathermap:141] >> rrd_start = '1212962400'
DEBUG: [@weathermap:141] >> rrd_aggregate_function = ''
...............
DEBUG: [wmrrd_read_from_real_rrdtool@WeatherMapDataSource_rrd.php:292] configs/mymap.cfg: RRD ReadData: traditional style
DEBUG: [wmrrd_read_from_real_rrdtool@WeatherMapDataSource_rrd.php:307] configs/mymap.cfg: RRD ReadData: Running: /usr/bin/rrdtool fetch /srv/www/cacti/rra/node1_traffic_in_1004.rrd AVERAGE --start 1212962400 --end start+120
DEBUG: [wmrrd_read_from_real_rrdtool@WeatherMapDataSource_rrd.php:327] configs/mymap.cfg: > 1212962460: 9.9756328170e+05 4.1407558058e+06
DEBUG: [wmrrd_read_from_real_rrdtool@WeatherMapDataSource_rrd.php:327] configs/mymap.cfg: > 1212962520: 1.5138245172e+06 4.3269857117e+06
DEBUG: [wmrrd_read_from_real_rrdtool@WeatherMapDataSource_rrd.php:327] configs/mymap.cfg: > DEBUG: [wmrrd_read_from_real_rrdtool@WeatherMapDataSource_rrd.php:335] configs/mymap.cfg: RRD ReadData: Read 3 lines from rrdtool
DEBUG: [wmrrd_read_from_real_rrdtool@WeatherMapDataSource_rrd.php:336] configs/mymap.cfg: RRD ReadData: Headings are: traffic_in traffic_out
...............
DEBUG: [DrawMap@Weathermap.class.php:2696] configs/mymap.cfg: Writing PNG file to mymap.png
DEBUG: [@weathermap:240] configs/mymap.cfg: Writing HTML to mymap.html
I have large rrd files with 90 days/1min rra´s.
July 22, 2008 - 11:06pm
I just edited the aggregation example to show the correct syntax, since I played with this for the first time in a while today for our live Cacti server.
August 6, 2008 - 6:35pm
It's not clear to me how the second cron job should look in order to pick up the new feature and build a different map from the same config file. Can you elaborate?
My apologies if this should be obvious, I just started using php-weathermap the day before yesterday. :)
--
Garry
August 7, 2008 - 10:42pm
Sure. You would do something like:
php weathermap --config my-map-config.cfg
normally.
So for the summary map, you can do
php weathermap --config my-map-config.cfg --define rrd_aggregate_function=95,PERCENT --define rrd_period= 2592000 --define rrd_start=now-1month
instead, to get a monthly 95th-percentile version of the same map. Make sense?
If you are using Cacti, you can do a similar thing, but in the web UI - add a second copy of the same map, and define the same 3 extra variables by clicking on the column that probably says 'standard' in the management screen.
August 6, 2008 - 11:43pm
Also, I think I may have found a bug. But it could just be that I'm not sure yet how the conf file directives all work together...
If I set the rrd_default_path, then later need to change something to use guage:, 'TARGET guage:example.rrd' will not work, due to 'guage:' being inserted in the path.
Easy way around this?
August 7, 2008 - 10:37pm
gauge: is spelt 'gauge' - it's not picking it up, so it assumes it's part of the filename.