Solar Irradiation Data

#   2-9   year month day hour
#  18-23  Global Horizontal Radiation
#  24-29  Direct Normal Radiation
#  30-35  Diffuse Horizontal Radiation
#  68-73  Dry Bulb Temperature
#  80-84  Relative Humidity
#  91-95  Wind Direction
#  96-100 Wind Speed

2 OzEA_DIRR0002

Alex Subject: BoM wind measurment stations, matched to solar stations Date: 2010-09-15 (at 12:38:24)

In order to reconcile potential wind farm output with potential solar farm output, wind speed BoM stations that match solar irradiance BoM stations have been gathered. In all except two cases it is the same site recording wind and solar, these two are highlighted with a bracketed comment.

BoM 31011 CAIRNS
BoM 26021 MT GAMBIER AIRPORT (The closest wind station)
BoM 5007 LEARMONTH
BoM 8051 GERALDTON
BoM 12038 KALGOORLIE-BOULDER
BoM 3003 BROOME
BoM 14015 DARWIN
BoM 72150 WAGGA
BoM 86282 MELBOURNE
BoM 91245 CAPE GRIM BAPS (may be the same site, but under a different name)
BoM 76031 MILDURA
BoM 15135 TENNANT CREEK
BoM 39083 ROCKHAMPTON
BoM 23034 ADELAIDE
BoM 15590 ALICE SPRINGS
BoM 200284 COCOS ISLAND

4 OzEA_DIRR0004

francis Subject: Looking at the BoM solar data Date: 2011-01-27 (at 22:27:57)

Some descriptive plotting of the BoM solar data (as provided; i.e. "raw") has been part of my familiarisation and tooling up approach. Actually scanning the data files by eye (as neatly formatted) is also a valuable thing to do. Here we look at distributions for the example case of Adelaide Airport. The data comes with quoted uncertainties, and we look at these also. Note that all zero and null values are ignored.

First, here are the distributions for the Direct (perpendicular to suns rays) radiation measurements:



In the third plot we examine the data on a more useful log scale; some care was required to make these traces (using a kernel density approach).


Second, here are the distributions for the Direct Horizontal radiation measurements (power flux through a horizontal square meter); the individual values are expected to be related to the previous Direct ones via the sine of the azimuth of the sun at the given time point:




Third, the Diffuse measurements (again as seen by a horizontal square meter):




And finally the Global measurements (expected to be, individually, the sum of the above Direct Horizontal and Diffuse values):




As a way of viewing all of the above in a single view, I have constructed the following panel plot:




For inspection and reference, here are such panel plots for all sixteen stations.

There are many minor observations I have made along the way... The observation of note is that the measurement errors (I think BoM call them uncertainties) are mostly quite modest but there are a small fraction of cases where they are large. Of particular note is the GERALDTON data where there are a significant number of large errors. From inspection of the data files it is seen that the larger error values, both in this case and the rest, are clustered together. Thus, the error statistics can be greatly improved, if need be, by culling particular blocks of data.

6 OzEA_DIRR0006

francis Subject: Processed BoM Solar Data Date: 2011-02-15 (at 21:18:04)

Here is the processed BoM Solar data, grouped into eight flat-files for each of the Direct, Diffuse, Direct_Horizontal and Global measurement types (and with measurements and uncertainties for each in their own file). The data has been time shifted to give the best correspondence to AEST (as per Comment 5) based on mean solar time. NaN values that correspond to night time (as established from previous data) are set to zero, but otherwise the NaN's have not been infilled. Will post on the NaNs shortly.

Here are the files:

OzEA_R1_BoM_Solar_Direct.flat.gz (2.1 MB) and uncertanties (1.1 MB)
OzEA_R1_BoM_Solar_Diffuse.flat.gz (2.0 MB) and uncertanties (0.7 MB)
OzEA_R1_BoM_Solar_Global.flat.gz (2.4 MB) and uncertanties (1.1 MB)
OzEA_R1_BoM_Solar_Direct_Horiz.flat.gz (2.0 MB) and uncertanties (0.9 MB)

8 OzEA_DIRR0008

francis Subject: missing solar data at Learmonth - an interesting example Date: 2011-02-17 (at 14:54:48)

was dealing with (ex) tropical cyclone Carlos ... and neglected to comment on my examination of the 2003-4-5 data. Considering only the Direct Radiation, and using the above described NaN analysis table, the major blocks of patchy data were noted and each examined by eye in the corresponding data file. Some are simply large chunks of missing data, others are more broken up. An example of missing data that has an interesting structure is that for Learmonth, both the 2003-11-19 to 2004-01-27 section and the 2004-11-16 to 2004-12-31 section. Might it be helpful to understand what has happened here? Could these missing values be infilled for the sake of an overall analysis, or would that be a stretch too far? I remain undecided and would appreciate comments, specific and generic, on handling missing data.

10 OzEA_DIRR0010

francis Subject: Solar Data Status for Round 1-a; just ground based data to start Date: 2011-03-03 (at 22:21:22)

so far as OzEA knows, ground based solar irradiance time series data for Australia is limited to the BoM data we have. Then there is the satellite data, in particular the 3tier data. Renewables SA have provided us with an example of this data, and you ask them first if you what the full set -- here we use *pieces of it* as needed.

for now we look at satellite data as a side game. Maybe (?) the Qld and NSW institutions can help with some sites around { Charleville, Roma, Bourke; etc }. There is also Innamincka and even Birdsville. Where would the transmission run? Multiple links perhaps: {into SA, to Alice; back to Brisbane; into western nsw}. What sort of storyline brings a Simpson Crossing to life?

12 OzEA_DIRR0012

francis Subject: BoM Hourly Solar Exposure Gridded Data Date: 2012-01-05 (at 11:13:16)

Following up on a paper from the AuSES Solar 2011 conference, I have found that BoM are (now) providing hourly gridded irradiance data in 5km cells from 1998 to 2010. See:

http://www.bom.gov.au/climate/how/newproducts/IDCJAD0111.shtml

Need some dollars to get a copy of this, so just noting here for reference, for now.

14 OzEA_DIRR0014

Francis Subject: The BoM gridded solar data - Part I Date: 2012-02-20 (at 17:51:06)

I've been holding the BoM gridded solar 'data' for a few weeks, after Ben Elliston (see above) kindly mailed it to me on a USB stick. First some overall comments, and then more specific ones.

Overall the Global Horizontal Irradiance (GHI) 'data' (estimated flux through a horizontal square meter) appears reasonable, and this data can be worked up to simulate PV farms just about anywhere in Oz, albeit at hour-level. The Direct Normal Irradiance (DNI) 'data' (direct rays through a square meter aligned to face the sun) is what we have for simulating Concentrating Solar Thermal (CST) farms (else be restricted as we have been to the ground based data). While, as below, there are some rough edges with the DNI gridded 'data', and these will be looked at further, I'm now happy enough that this 'data' is useable.

Site Selection Project: as for the wind data, we want to identify specific sites around the country, perhaps around 100, where we will simulate CST or PV farms. Then I will batch process time series for these sites, and do the sanity checking and characterisation. Some sites will be for consideration in OzEA scenarios, some will be where existing PV with available output data are situated, and others will simply be of interest to community members for some reason. Get your orders in! [for now either comment below or mail me directly with details].


Background on the BoM gridded data: First, here for GHI, and here for DNI, is the metadata provided by BoM (I have converted from doc to pdf). In short, one or another satellites snaps the continent every hour (0.5 to 1.0 µm). These images are processed to obtain GHI estimates on a ~5 km grid through a model described in:

Weymouth G.T. and Le Marshall J.F. 2001. Estimate of daily surface solar exposure using GMS-5 stretched-VISSR observations. The system and basic results. Aust. Meteor. Mag., 50, 263-278

A further model is used to obtain DNI estimates as described (behind a pay-wall) in:

Ridley B., Boland J. and Lauret P. (2010). Modelling of diffuse solar fraction with multiple predictors. Ren. Energy, 35, 478-483

[I take it that by modelling the diffuse "fraction", what remains of the GHI is the direct part, leaving some geometry to obtain the DNI estimates]

Because these GHI and DNI estimates are the output of models, and model fitting against ground based data, I use the word 'data' gingerly. With this point made, I drop the quotes.


The data is voluminous, with a 679 rows x 839 columns data file for each hour under consideration (~6500 files per year, with middle-of-the-night hours not given). Here is an example BoM gridded data file. Individual values are in Watts per square meter. The NODATA (Null) value used by BoM is -999. Lower Left corner is 112.025 E, -43.925 S, and the grid size is 0.05 degrees (roughly 5km, although this will vary with location and direction). Having so many individual files makes the data handling somewhat painful, but that's all water under the bridge now. I have automated the process of extracting a time series for a location, or set of locations.

In processing the data for OzEA I have infilled with '0' for the not given night-time hours (those that are given have -999). I've also used 'NaN' for the small number of cases where there were missed daytime hours in the BoM data. Thus, in viewing the data these different 'null' value forms indicate their origin.

The BoM data corresponds to Universal Time (UT), which is the new GMT, although it turns out (see metadata) that the satellite images are mostly taken in the latter part of the hour. To maintain consistency with the OzEA solar data time base (AEST, value representing the previous hour, or half hour), the time base is adjusted +11 hours from the UT given (in the individual data filenames). That is, +10 hr for UT -> AEST, and +1 hr for the interpretation of what has happened, rather than what happens later in the specified hour. As seen in what follows, this gives the best alignment with the ground based data (see also comment #5 above for time base issues in respect to the ground based data).


Comparison with the Ground Based measurements: Choosing 2007 as a reference year, and referring to the ground based data coverage analysis (see comment #7 above), gives us 8 reference sites: BROOME AIRPORT, DARWIN AIRPORT, WAGGA WAGGA AMO, MELBOURNE AIRPORT, CAPE GRIM RADIATION, ROCKHAMPTON AERO, ADELAIDE AIRPORT, and ALICE SPRINGS AIRPORT.

Note, before we proceed further, that this comparison work is NOT a validation of the gridded data, as the ground based data is used to parameritise the models. It IS two things: it provides validation of the data munging work done here (to get the correct values in the correct spots); and it provides an opportunity to look at the data -- a habit that OzEA commends as an important, albeit time consuming, step in empirical endeavours.

The alignment of the reference sites (ground based measurements) with gridded data for aligned grid cells is given in the following two text files:

* GHI comparison [1.5 MB]
* DNI comparison [1.5 MB]

These files are for viewing in a basic text editor (fixed width characters; no wrap). I've spend something like ten hours eyeballing these files (and their developmental counterparts).

First up was to see how well the mapped grid positions aligned (see positions in the files). To start Broome and Darwin end up 'in the water' (as signified by all NODATA values); moving Broome two steps east, or one east and one north, retrieved the land, and this is consistent with the geography (using Google earth, and with Broome airport close to the water); for Darwin need one step either south or east, which is again consistent with the geography. With the remaining four of six sites that are coastal, I examined the geography and explored grid point moves that would put the site just in, or just not in, the water. Overall, I did not find any obvious systematic biases, with all observations broadly consistent with the geography (as observed using Google Earth), perhaps suggesting a positional accuracy at around the one grid cell level.

Note that it may be unfortunate that six from eight of the reference sites are close-costal, where the combination of rapidly changing terrain with urban or semi-urban land use can (I speculate) be more difficult for the computational analysis of ground albedo from the satellite images than for more texturally even terrain found inland.

Next, with Broome and Darwin moved ashore, I scripted up the comparison files given above (and resolved the time-base issues as discussed). From 'eyeballing' these files I've made many observations, and here focus on a few general issues.

* Individual values can be way out (e.g. 916 vs 63 W/m2 for Broome GHI on 2007-03-15), however this sort of discrepancy is the exception. In general the differences are modest. As discussed in the literature, the gridded data is more accurate in clear sunny conditions, and less accurate in more overcast conditions (but still broadly pretty good from my observations).

* For both the GHI and DNI data, the time base appears to shift out by up to half an hour at times, although this issue is expected to reside with the ground based data (see comment #5) rather than the gridded data.

* The GHI data overall appears very reasonable. The tails at either end of the day are not well captured by the gridded data, but as these values are low (<200) this does not matter much in terms of simulating PV installations.

* For the DNI data overall, the issues with the consistency of values at the beginning and end of the solar day are more pronounced. This may be an issue of some concern when using this data for simulating CST farms, and this is an issue that I will return to explore further and quantitatively.


Concluding Comment
While there is more work to do characterising the gridded solar data, especially in relation to values at the beginning and end of the day, it is also time to get on with identifying sites around Australia where OzEA and other interested parties want to establish individual solar irradiance time series from the gridded data, either for consideration in scenarios and/or because of alignment with existing solar PV (and associated power production data).