Data logging with open source hardware and software

in the energy sector

Solar panels, Arduino, PHP, MySQL, and Flotr
Colin Beckingham (colbec@start.ca)
Researcher
Freelance

19 July 2011

Explore how to monitor simple climatic conditions with the Arduino in the context of solar power
generation. Specifically, get an introduction to basic temperature sensing with a TMP36 chip
and light with a light-emitting diode (LED), and store the information for later retrieval. Then
smooth out selected results and display in a chart, using PHP as a data processor, MySQL as
storage, and Flotr as a charting library.

Overview
As power generation moves towards cleaner and smarter sources, small-scale photovoltaic panel
arrays are springing up on roofs and in backyards. Some of these installations are sophisticated
with a high level of monitoring and self-adjustment; others have no self-awareness. In the latter
case, it is still helpful for the owner to have a good and inexpensive source of basic operating data.
You want to know if your panels are working optimally given the conditions. Both light and
temperature, with the addition of wind speed and other factors, can influence panel output. More
light generates more power, but as more power is generated the cells heat up, which reduces their
efficiency. Airflow helps to dissipate heat. Ideally, you want bright light and cool conditions, and a
good breeze or other artificial cooling.
The Arduino acts as an intermediary between the sensors and your data storage. It is a
straightforward matter to make a permanent record and display the data as required. This article
examines data logging of temperature using the TMP36 temperature sensor and a simple lightemitting diode (LED) for light.

Arduino
The Arduino is an inexpensive, adaptable, and programmable open source microprocessor that
can read data input in the form of voltage at its analog pins. See Resources for basic information
about the unit and a good developerWorks introduction to this unit in a game context. With sensors
Copyright IBM Corporation 2011
Data logging with open source hardware and software in the
energy sector

Trademarks
Page 1 of 14

developerWorks

ibm.com/developerWorks/

connected to specific input pins, the unit programmatically reads the data at those pins. What it
does with that information depends on how you configure the hardware.
The simplest way of getting the data from the Arduino is with the unit connected directly to a host
computer Universal Serial Bus (USB) interface, reading the data as if it were a serial connection.
However, the Arduino can also act independently of a computer given a power supply and an
alternate communications channel. You can add extension boards ("shields") to the Arduino to
store data directly to a microSD memory card, transmit data over a cat5e network cable through
mini web server, or even transmit data wireless to a compatible receiver.
Figure 1 shows an Arduino microprocessor with an Ethernet shield mounted and a cat5 Ethernet
and power supply cables attached, which gives you an idea of the size of the unit.

Figure 1. Arduino with Ethernet shield

Some sensors have their own data smoothing internal circuitry. The simpler ones do not.
Therefore, the question arises whether the Arduino should do any necessary processing before
storage or transmission, or should data simply be reported raw as observed for later processing on
another machine. Either is perfectly acceptable. This project simply reports the raw observations
for later processing. This gives the option to change the algorithm for smoothing at any time if
necessary. When a smoothing process is applied, it is baked in and probably not reversible. In a
real-time data situation where the smoothing requirements are well understood, it makes more
sense to have the Arduino do the smoothing.

Sensors
You can get a wide variety of sensors to work with the Arduino. Here, the TMP36 and an LED
provide a simple, inexpensive, and easily repeatable alternative.
Data logging with open source hardware and software in the
energy sector

Page 2 of 14

ibm.com/developerWorks/

developerWorks

The TMP36 is a specially constructed transistor. Give it a voltage and it returns another voltage
that varies consistently according to the ambient temperature on a different combination of
connectors. Record the output voltage and perform some simple arithmetic to find the temperature
in degrees Celsius or Fahrenheit according to preference. See Resources for help on both the chip
and how to connect it to the Arduino.
It seems simple enough to log temperature data in a solar generation context, but there are a
couple hurdles to surmount. The first issue, related to ambient temperature, is determining where
to locate the sensor. To avoid false readings you want to place it out of direct sunlight and in a
weatherproof location that provides enough ventilation. Secondly, a solar panel consists of a
number of non-identical cells, each of which runs at its own temperature in the same ambient
conditions. In ideal circumstances, you measure all the cells and take the average. An alternate is
to try a number of them to identify a representative average cell and monitor it alone.
An LED can both light up when given current at a suitable voltage and generate a voltage when
exposed to light (see Mike Cook's article in Resources). The greater the intensity of the light, the
more voltage the Arduino sees at the connected pin. Unlike the temperature sensor, which is
designed to be mappable into a specific temperature scale, you have to experiment to see how the
output from your LED translates to light intensity on a known scale.

Practical setup
Imagine that your solar array location has power, no Ethernet local area network (LAN) jack to
plug into, but is within range of a wireless access point. Then the local power can fire up both the
access point and the Arduino, with the Arduino plus Ethernet shield plugged into the repeater.
You then have analog pins 3 to 6 available for sensor input; assuming pins 1 and 2 are reserved
for use by the Ethernet shield (see Resources for more about pin usage). Four pins means you
can have four sensors, say two temperature sensors, one for ambient and the other for panel
temperature, a light sensor, and a wind speed sensor. This is a rich set of data for a passive panel
setup.
You can achieve the actual wiring between the Arduino and the sensors over short distances with
regular cat3 telephone cable, which is an unshielded bundle of two pairs of 24 American wire
gauge (AWG) wire. This also gives you the option of using readily available telephone adapters
and cabling.
Working with the Arduino and Ethernet shield combination is not as easy as an Arduino that is
USB/serial connected alone. Here are some details to consider:
Normally you don't poll voltages at pins that have no sensors connected to them.
Through a barrel connector, the Arduino can accept power at a range of different voltages
(see Resources for details). Analog pin voltage readings seem to depend critically on the
voltage of the power supply to the Arduino. Make sure that you do your calibration of sensors
with the same power supply unit that you intend to use in the field. Advanced users make use
of the Analog Reference (AREF) pin to provide a reference voltage.
Data logging with open source hardware and software in the
energy sector

Page 3 of 14

developerWorks

ibm.com/developerWorks/

Attach your temperature sensor to the back of a solar cell. Duct tape seems to work quite
well to hold the sensor against the cell even in warm conditions over the short term, but make
sure that any exposed wires or connectors are both protected and insulated by good electrical
tape. It is not unusual for a cell to rise to a normal operating cell temperature (NOCT) of 50
degrees C (124 F) in bright light when the ambient temperature is 20 degrees C (69 F) and
there is a steady breeze blowing.
The Arduino with Ethernet shield is quite power hungry. If you use a set of fully charged AA
size batteries to provide voltage and current they last only a few hours. Advanced users look
at the sleep modes available to control power usage.
Figure 2 shows TMP36 sensors attached to a solar rack and panel. On the right, under black
electrical tape, is a sensor for ambient temperature taken from the metal pole. To the left is a
second sensor attached to the back of a solar cell with duct tape. The duct tape retains adhesion a
lot better than the consumer-grade electrical tape, at least in the short term.

Figure 2. Sensors on panel

Figure 3 shows a green LED inside an inverted test tube for protection against rain.

Data logging with open source hardware and software in the

energy sector

Page 4 of 14

ibm.com/developerWorks/

developerWorks

Figure 3. Light sensor

The Arduino program (or sketch, as Arduino calls it) that allows access to the data collected is a
simple web server that you access with a browser or other script (see Listing 1). The basic code is
provided by Arduino as part of the examples collection and requires some editing to adapt to your
own requirements (see Resources).

Listing 1. Arduino web server

// based on example web server sketch from http://arduino.cc/
#include <SPI.h>
#include <Ethernet.h>
//
byte mac[] = { 0x!!, 0x!!, 0x!!, 0x!!, 0x!!, 0x!! };
byte ip[] = { 192,168,0,xxx };
Server server(80);
//
void setup()
{
// start the Ethernet connection and the server:
delay(10000);
Ethernet.begin(mac, ip);
server.begin();
}
//
void loop()
{
// listen for incoming clients
Client client = server.available();
if (client) {
// an http request ends with a blank line
boolean currentLineIsBlank = true;
while (client.connected()) {
if (client.available()) {
char c = client.read();
// if you've gotten to the end of the line (received a newline
// character) and the line is blank, the http request has ended,
// so you can send a reply
if (c == '\n' && currentLineIsBlank) {
// send a standard http response header
client.println("HTTP/1.1 200 OK");

Data logging with open source hardware and software in the

energy sector

Page 5 of 14

developerWorks

ibm.com/developerWorks/

client.println("Content-Type: text/html");
client.println();
//
// output the value of each analog input pin
for (int analogChannel = 3; analogChannel < 7; analogChannel++) {
//client.print("analog input ");
client.print(analogChannel);
client.print(" : ");
client.print(analogRead(analogChannel));
client.println("<br />");
}
break;
}
if (c == '\n') {
// you're starting a new line
currentLineIsBlank = true;
}
else if (c != '\r') {
// you've gotten a character on the current line
currentLineIsBlank = false;
}
}
}
// give the web browser time to receive the data
delay(1);
// close the connection:
client.stop();
}
}

Listing 1 begins by specifying the header libraries needed, the Media Access Control (MAC)
address of the Ethernet shield, and the Internet Protocol (IP) address by which you intend to know
the Arduino. It also sets up a server object and a variable used to store a value that can be later
discarded. The setup() subroutine runs just once on startup and initializes the server. The loop()
subroutine then loops forever, waiting for a Hypertext Transfer Protocol (HTTP) request and then
answering with the data needed. In response to a request, it sends a standard HTTP response
header, collects readings from the relevant pins, and sends the data package in text format. In this
case, the data provided is the voltage experienced at the instant of the request on analog pins 3,
4, 5, and 6. No data is sent for pins 1 and 2 because they are used by the interface between the
Arduino board and the Ethernet shield. Change the range of channels to read only the pins you are
using.
The response will be something like the output shown in Listing 2.

Listing 2. Arduino response

3
4
5
6

:
:
:
:

292
288
286
280

This gives the number of the pin and the value from a scale of 0 to 1023, the two numbers
separated by a colon. The output can be more specific than just specifying the analog pin number,
but the generic number allows flexibility because it does not tie down any one pin to any one job.

Data logging with open source hardware and software in the

energy sector

Page 6 of 14

ibm.com/developerWorks/

developerWorks

Data storage
The next thing to consider is how the data is stored. In this case, you are using MySQL as the
back end, as shown in Listing 3.

Listing 3. Back-end table

CREATE TABLE IF NOT EXISTS `readings` (
`readid` int(11) NOT NULL AUTO_INCREMENT,
`tstamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
`pin` int(11) NOT NULL,
`value` float NOT NULL,
PRIMARY KEY (`readid`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 AUTO_INCREMENT=1 ;

This code creates a table with an ID field, the timestamp defaulting to the system time of the entry,
the pin the reading relates to, and a field for the actual reading, which in this case is stored as a
float.

Data collection and smoothing

With the back end defined, you can now use PHP to query the Arduino, analyze the response, and
store the observations in the back end, as shown in Listing 4.

Listing 4. PHP data logging script

<?php
// reader for panel monitors
$fp = fopen("http://192.168.0.xxx","r");
$mysqli = new mysqli("$server",$username,$password,$mysolar);
while ($line = fgets($fp,32)) {
$line = str_replace("
","",trim($line));
$key = substr($line,0,1);
switch ($key) {
case 3:
// things to do
break;
case 4:
// things to do
break;
case 5:
// this is light from LED recorded as is
$val = (int) substr($line,4);
$sql = "insert into readings values(NULL,NULL,$key,$val)";
$result = $mysqli->query($sql);
//echo "val = $val\n";
break;
case 6:
// temperature from a TMP36 attached to back of solar cell
$val = (int) substr($line,4);
$tmp = degc($val); // convert to Celsius/Centigrade
$tmp -= 5; // calibration
$sql = "insert into readings values(NULL,NULL,$key,$tmp)";
$result = $mysqli->query($sql);
break;
default:
//echo "Found strange key $key!\n";
break;
}
}
function degc($v) {
$t = $v * (5/1024);

Data logging with open source hardware and software in the

energy sector

Page 7 of 14

developerWorks

ibm.com/developerWorks/

$t -= 0.5;
$t *= 100;
return round($t,1);
}
?>

This script begins by opening the connection to the Arduino in a file pointer for reading. Then it
opens the connection to the MySQL back end for data output. It then waits for the response from
the Arduino and goes into a loop to read lines reported. Each of the four anticipated lines contains
a Hypertext Markup Language (HTML) break tag, which makes it easier to read the output in a
browser while testing. The script removes this break. The resulting lines begin with the number
of the pin followed by the voltage found. Focusing on the pin number, it then goes into a switch
structure that takes a different action depending on which sensor is connected to the pin.
Calibration takes place in this step. That is, if you find that a sensor consistently overestimates by
a certain value, you can subtract that value before the final value is reported. There is a specific
function for converting the measured millivolts from the TMP36 to degrees Celsius, changing from
a range of 0-1023 to 1 to 5, finding the zero point and scaling the result by 100. Lastly, it stores the
cleaned up and rounded value in the database. How do you know if your sensor needs correction?
One way is to measure a cell that behaves similarly with a point and shoot emissivity temperature
reader.
The insert SQL statement starts with two NULL values, allowing the back end to substitute an
auto-increment ID number for the first and the current timestamp for the second. It then plugs in
the number of the pin and the reading for that pin. You can use this script from your crontab list for
regular processing or run as required from the command line.

Chart display
With the back end filling up with data, the next task is to make the data easily readable. There are
a number of charting libraries available, each with its own strengths and weaknesses. Flotr (see
Resources) is one example that uses JavaScript to display a chart in a browser window. The code
shown in Listing 5 reads from the data stored in the readings table and presents the temperature
and light data in separate charts.

Listing 5. PHP chart generator using Flotr

<?php
// read solar data and display in flotr chart
$doctype = "<!DOCTYPE html PUBLIC '-//W3C//DTD XHTML 1.0 Transitional//EN'
'http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd'>
<html xmlns='http://www.w3.org/1999/xhtml' xml:lang='en'>
<head>
<title>LOG reader</title>
<meta http-equiv='Content-Type' content='text/html;charset=UTF-8' />
<script language=\"javascript\" type=\"text/javascript\"
src=\"../flotr/lib/prototype.js\"></script>
<script language=\"javascript\" type=\"text/javascript\"
src=\"../flotr/lib/base64.js\"></script>
<script language=\"javascript\" type=\"text/javascript\"
src=\"../flotr/lib/canvas2image.js\"></script>
<script language=\"javascript\" type=\"text/javascript\"
src=\"../flotr/lib/canvastext.js\"></script>
<script language=\"javascript\" type=\"text/javascript\"

Data logging with open source hardware and software in the

energy sector

Page 8 of 14

ibm.com/developerWorks/

developerWorks

src=\"../flotr/flotr.js\"></script>
</head><body>";
$mysqli = new mysqli($server,$username,$password,$dbname);
// get temp and light data and chart it
$sensors = array('Temp'=>6,"Light"=>5);
foreach ($sensors as $label=>$sensor) {
$sql = "select tstamp,value from readings
where pin=$sensor order by tstamp asc";
$result = $mysqli->query($sql) or die($mysqli->error);
$mydata1 = "[ ";
while ($row = $result->fetch_array()) {
$cfdoy = $row[0];
$cfdoy = (strtotime($cfdoy)-(5*60*60))*1000;
$cfamt = $row[1];
$mydata1 .= "[ $cfdoy , $cfamt ] ,";
}
$mydata1 = substr($mydata1,0,-2)." ]";
$leg = ($sensor == 18) ? "dC" : "mV";
$title = ($sensor == 18) ? "Cell Temperature" : "Light";
$conth .= "<div>$label</div>\n
<div id=\"container$sensor\" style='width:600px; height:250px;'>
<script type='text/javascript'>
var f = Flotr.draw(
$('container$sensor'), [
{ // => first series
data: ".$mydata1.",
label: '$leg',
htmlText: false,
lines: {show: true}
}],
{
xaxis: {
title: 'Time',
mode:'time',
noTicks: 10,
labelsAngle:45
},
yaxis: {
title: '$leg',
noTicks: 8
},
title: '$title',
selection: {
mode: 'x',
color: '#B6D9FF',
fps: 20
},
mouse: {
track: true,
relative: true,
margin: 5,
trackFormatter: function(obj) {
var dd = parseInt(obj.x);
var d = new Date(dd);
return (d.getHours()+4) + ':'
+ d.getMinutes() + ' | ' + obj.y;
},
position: 'sw',
lineColor: '#FF3F19',
trackDecimals: 1,
sensibility: 2,
fillOpacity: 0.4
}
}
);
</script></div>";
}

Data logging with open source hardware and software in the

energy sector

Page 9 of 14

developerWorks

ibm.com/developerWorks/

echo "$doctype"."<html><body>".$conth."</body></html>";
?>

The code in Listing 5 begins by defining a string that is the start of the Extensible Hypertext
Markup Language (XHTML) output, including the references to the JavaScript Flotr libraries.
Then it opens a connection to the MySQL server where the data is stored and sets up an array
containing the numbers of the pins used, steps through the array elements fetching data for the
relevant sensor and substituting it into the JavaScript script.
The script is set up to treat X axis data as time values on the 24-hour clock for both charts and
contains mouse instructions that allow you to hover over the produced line and display the related
coordinates.
During the display phase, you might think about smoothing. This can be the use of a moving
average or other weight-based scheme that moderates extreme values.
Using real data generated from this setup sampled every five minutes, the charting script produces
the output shown in Figure 4 with no smoothing. The conditions were sunny in May in Ontario, with
occasional clouds. You can get a copy of the data from the Download section.

Figure 4. Example output

Note the close correlation between the light levels and cell temperature, which is to be expected.

Data logging with open source hardware and software in the

energy sector

Page 10 of 14

ibm.com/developerWorks/

developerWorks

Conclusion
The open source Arduino is useful as a simple and adaptable processor for sensor data in a solar
generation context. A wide variety of sensors are available, and as long as a sensor produces
a suitable output voltage, or alternatively a digital output, that can be cleanly registered by the
Arduino, it can collect and report that data. The TMP36 and LED as sensors are about as basic as
you can getserious users will upgrade from these alternatives.
Good numbers for the behavior of panels in varying conditions can be helpful to managers of
photovoltaic units because, given the data record, they can form a baseline of observations.
Managers can then make changes according to reasoning and advice and then assess whether
later panel manipulation has had a positive effect or not.

Data logging with open source hardware and software in the

energy sector

Page 11 of 14

developerWorks

ibm.com/developerWorks/

Downloads
Description

Name

Size

Data sample

readings.sql.zip

1KB

Data logging with open source hardware and software in the

energy sector

Page 12 of 14

ibm.com/developerWorks/

developerWorks

Resources
Learn
"Building an Arduino-based laser game": (Duane O'Brien, developerWorks, Dec 2008) Read
a good introduction to Arduino, and find instructions on how to set up an LED as a light
detector.
LED Sensing: Find out more about the physics behind how and why LEDs can act as light
sensors.
Arduino Forum: Learn more about Arduino from other users.
Arduino tutorial: Learn more about setting up an Arduino from this tutorial that includes
suggestions on an external power supply.
Arduino Shield List pin usage: Check out pin usage details for 244 shields from 107 makers,
and counting.
Arduino and TMP36: Get detailed instructions on connecting Arduino and TMP36.
Arduino as a web server: Follow the Arduino tutorial to use your Ethernet Shield and your
Arduino to create a simple web server.
Flotr: Find information and examples related to the Flotr JavaScript charting library.
TMP36, voltage output temperature sensor: Get detailed technical information the sensor.
developerWorks PHP content Browse the PHP content on our site.
developerWorks on Twitter: Follow us.
PHP project resources: Expand your PHP skills.
Events of interest: Check out upcoming conferences, trade shows, and webcasts that are of
interest to IBM open source developers.
developerWorks Open source zone: Find extensive how-to information, tools, and project
updates to help you develop with open source technologies and use them with IBM's
products.
Get products and technologies
Arduino: Visit the main site.
MySQL community edition: Access a freely downloadable version fo this open source
database.
PHP: Check out the latest release of this general-purpose scripting language.
IBM trial software: Innovate your next open source development project using trial software,
available for download or on DVD.
Discuss
developerWorks community: Connect with other developerWorks users while exploring the
developer-driven blogs, forums, groups, and wikis. Help build the Real world open source
group in the developerWorks community.

Data logging with open source hardware and software in the

energy sector

Page 13 of 14

developerWorks

ibm.com/developerWorks/

About the author

Colin Beckingham
Colin Beckingham is a freelance researcher, writer, and programmer who lives in
eastern Ontario, Canada. Holding degrees from Queen's University, Kingston, and
the University of Windsor, he has worked in a rich variety of fields including banking,
horticulture, horse racing, teaching, civil service, retail, and travel and tourism. The
author of database applications and numerous newspaper, magazine, and online
articles, his research interests include open source programming, VoIP, and voicecontrol applications on Linux. You can reach Colin at colbec@start.ca.
Copyright IBM Corporation 2011
(www.ibm.com/legal/copytrade.shtml)
Trademarks
(www.ibm.com/developerworks/ibm/trademarks/)

Data logging with open source hardware and software in the

energy sector

Page 14 of 14