Wednesday, August 30, 2017

RF Noise Monitoring system for HF

RF Noise Monitoring system on a Raspberry Pi

(This system will record RF signals and upload the results to the Server where trending graphs are created)

This system software is available pre installed on a 16GB SD card from Giga Technology
Raspberry Pi OS with HF monitoring software pre installed
What do I get on the software?
I system that will record 1Mhz band with samples every 2 minutes generating Graphs and Heat-map of the hole 1 to 30Mhz band.
What hardware do I need?
The  100khz to 1.7Ghz HF dongle as in picture below Screenshot of Graph monitoring.
url of link on system http://localhost/hf_noise/graph/1Mhz_Power.php
Link to live system is here http://zr6aic.giga.co.za/hf_noise/graph/1Mhz_Power.php 
Screenshot of the HF Monitoring system

How does this system work?

Block Diagram of the HF Noise Monitoring system

How dose the HF Noise Monitoring system work

  • The HF noise monitoring system takes a 1Mhz bandwidth samples every 2 minutes using the rtl_power utility and save the measurements in a CVS fie.
  • The 2 minute scheduling is done with a Crontab calling a script in the hf_noise directory.
  • The RF Samples is taken at a 1Mhz bandwidth from 1M to 30Mhz therefore 29 of  the CVS files is created and gets appended as the measurements is made.
  • A Perl script utility is the executed from a cron scedular  to read the 29 CSV files and import there data into a RRD database.
  • The RRD database then gets interrogated by another Perl script scheduled by a cronjob to generate the graph Images and and html files and get saved in the Apache web server graph directory witch is the exposed on http://localhost/hf_noise/graph/1Mhz_Power.php
  • Link to Live system http://zr6aic.giga.co.za/hf_noise/graph/1Mhz_Power.php 
  • The RRD database will then create the Hourly, Daily,Weekly,Monthly and Yearly averages and   then generates the graphs every 2 minutes according to the parameters passed to Database.
  • The generated graphs is then displayed in a php page via a Apache web server with an php plugin on the following url http://localhost/hf_noise/graph/1Mhz_Power.php.
  • Trends can then be seen on a graphs over time.
  • The heat map is generated by a python script from different CSV files and is saved in the /var/www/hf_noise/images directory and can be accessed on the local web server on the following url http://localhost/hf_noise/images
  • Link to live system http://zr6aic.giga.co.za/vhf_heatmap/images/
  • The systems has a static IP by default on Ethernet 192.168.10.200 and 192.168.10.205 on wifi if you have wifi dongle installed. You can change it to your ip

Here is a list of RRD database management tips

  • How to backup my RRD db data?
  • use rrdtool dump hf_noise.rrd > filename.xml to export the data to XML format.
  • How to Restore my RRD db data?
  • Transfer the XML dump to the target system. Run rrdtool restore [--range-check|-r] [--force-overwrite|-f] filename.xml filename.rrd to create a new RRD from the XML dump. See rrdrestore for details.
  • How to change some of the the RRD db data?
  • Use rrdtool dump to export RRD files to XML.
    Open the XML file, find and edit the bad data.
    Restore the RRD file using rrdtool restore.

 How to generate a brand new RRD db?


There is an script in the hf_noise directory called create_rrd_db.sh
You can just run this script in the hf_directory sudo ./create_rrd_db.sh

Here is the details of the scrip.
    rrdtool create  hf_noise.rrd --start now-2d --step 120 DS:1Mhz_Power:GAUGE:120:-40:10 DS:2Mhz_Power:GAUGE:120:-40:10 DS:3Mhz_Power:GAUGE:120:-40:10 DS:4Mhz_Power:GAUGE:120:-40:10 DS:5Mhz_Power:GAUGE:120:-40:10 DS:6Mhz_Power:GAUGE:120:-40:10 DS:7Mhz_Power:GAUGE:120:-40:10 DS:8Mhz_Power:GAUGE:120:-40:10 DS:9Mhz_Power:GAUGE:120:-40:10 DS:10Mhz_Power:GAUGE:120:-40:10 DS:11Mhz_Power:GAUGE:120:-40:10 DS:12Mhz_Power:GAUGE:120:-40:10 DS:13Mhz_Power:GAUGE:120:-40:10 DS:14Mhz_Power:GAUGE:120:-40:10 DS:15Mhz_Power:GAUGE:120:-40:10 DS:16Mhz_Power:GAUGE:120:-40:10 DS:17Mhz_Power:GAUGE:120:-40:10 DS:18Mhz_Power:GAUGE:120:-40:10 DS:19Mhz_Power:GAUGE:120:-40:10 DS:20Mhz_Power:GAUGE:120:-40:10 DS:21Mhz_Power:GAUGE:120:-40:10 DS:22Mhz_Power:GAUGE:120:-40:10 DS:23Mhz_Power:GAUGE:120:-40:10 DS:24Mhz_Power:GAUGE:120:-40:10 DS:25Mhz_Power:GAUGE:120:-40:10 DS:26Mhz_Power:GAUGE:120:-40:10 DS:27Mhz_Power:GAUGE:120:-40:10 DS:28Mhz_Power:GAUGE:120:-40:10 DS:29Mhz_Power:GAUGE:120:-40:10 DS:30Mhz_Power:GAUGE:120:-40:10 RRA:AVERAGE:0.5:1:864000 RRA:AVERAGE:0.5:60:129600 RRA:AVERAGE:0.5:3600:13392 RRA:AVERAGE:0.5:86400:3660

I have still got my csv files and want to reload it into my rrd database

Still need to add details how to do this.





HF Heat map monitoring System. 


Link on local network http://localhost/hf_noise/images

link to live system http://zr6aic.giga.co.za/hf_noise/images/
Here is  a screenshot of the monitoring page and the green button
 can be used to select the frequency band of interest


Single view

Here is a view of the HF spectrum from 1Mhz to 30Mhz (activity can now be seen for the day)
Here is a view of the heat map that was selected
If you click on the image the the image will zoomed to maximum

Enlarged View

Here is a screenshot of the enlarged view.

So here is an example of the utility that generate the CSV file

Noise Recording

rtl_power -f 24M:1.7G:1M -g 50 -i 20m -1 noise-unmodded.csv

Heat map Plotting.

Still need to add detail


Installing the system from scratch (not yet complete)

sudo apt-get install cpanminus

sudo cpanm Text::Trim
sudo cpanm  Text::CSV
sudo cpanm  Date::Manip

sudo cpanm Text::Trim qw(trim)
sudo cpanm RRD::Simple ()

rrdtool update hf_noise.rrd ds-name:30Mhz_Power $recordtime:$rrdfields[29]\n";


rrdtool create  hf_noise.rrd --start now-2d --step 120 DS:1Mhz_Power:GAUGE:120:-40:10 DS:2Mhz_Power:GAUGE:120:-40:10 DS:3Mhz_Power:GAUGE:120:-40:10 DS:4Mhz_Power:GAUGE:120:-40:10 DS:5Mhz_Power:GAUGE:120:-40:10 DS:6Mhz_Power:GAUGE:120:-40:10 DS:7Mhz_Power:GAUGE:120:-40:10 DS:8Mhz_Power:GAUGE:120:-40:10 DS:9Mhz_Power:GAUGE:120:-40:10 DS:10Mhz_Power:GAUGE:120:-40:10 DS:11Mhz_Power:GAUGE:120:-40:10 DS:12Mhz_Power:GAUGE:120:-40:10 DS:13Mhz_Power:GAUGE:120:-40:10 DS:14Mhz_Power:GAUGE:120:-40:10 DS:15Mhz_Power:GAUGE:120:-40:10 DS:16Mhz_Power:GAUGE:120:-40:10 DS:17Mhz_Power:GAUGE:120:-40:10 DS:18Mhz_Power:GAUGE:120:-40:10 DS:19Mhz_Power:GAUGE:120:-40:10 DS:20Mhz_Power:GAUGE:120:-40:10 DS:21Mhz_Power:GAUGE:120:-40:10 DS:22Mhz_Power:GAUGE:120:-40:10 DS:23Mhz_Power:GAUGE:120:-40:10 DS:24Mhz_Power:GAUGE:120:-40:10 DS:25Mhz_Power:GAUGE:120:-40:10 DS:26Mhz_Power:GAUGE:120:-40:10 DS:27Mhz_Power:GAUGE:120:-40:10 DS:28Mhz_Power:GAUGE:120:-40:10 DS:29Mhz_Power:GAUGE:120:-40:10 DS:30Mhz_Power:GAUGE:120:-40:10 RRA:AVERAGE:0.5:1:864000 RRA:AVERAGE:0.5:60:129600 RRA:AVERAGE:0.5:3600:13392 RRA:AVERAGE:0.5:86400:3660


sudo apt-get install librrds-perl
sudo apt-get install rrdtool
perl -MCPAN -e 'install Test::Deep'
perl -MCPAN -e 'install Test::Pod'
perl -MCPAN -e 'install Test::Pod::Coverage'
perl -MCPAN -e 'install RRD::Simple'

Still need to add details





Here is a video where the reason for the Noise monitoring system gets explained and was presented at the AMSATSA HF Workshop.