HOME

OZ-ENERGY-ANALYSIS.ORG   -   open science for the new millennium

THE STORIES   |   DATA   |   ANALYSIS   |   MODELS   |   LITERATURE   |   DISCUSSIONS

Last Editted (minor tidy up): 7th Jan 2012

OPEN SCIENCE

What is Open Science?

There is a simple answer... and then there is a more nebulous but ambitious vision.

Preamble

Science is special because of its capacity to uncover objective truth. Of course, individual scientists are people too, with a world of subjective, intuitive and more-or-less objective reasons for examining certain questions using chosen methods. While the 'who' and the 'why' can be interesting, especially historically, they do not matter much when judging scientific claims. Scientific conclusions (ideally) stand or fall based on the evidence for and against their correctness.

The process of Science depends very much on the assessments of peers. Often this works well, especially given the expert knowledge and specialised experience that can be needed to understand and contextualise technical work. At other times the peer system resembles a court in which the defendant judges the plaintiff. Sometimes the prevailing wisdom of the establishment, intellectual fashions, collegiate nepotism, or any number of human subjectivities can retard the clear-eyed assessment of correctness. So it has always been.

In modern times the Science professions have changed radically. It is said there are more 'scientists' active today than have ever practised prior to the 20th Century, and the enterprise continues to expand. We no longer work with seven or a dozen laboriously collected data points against which we fit carefully considered models by hand; rather, we are awash with data and computational tools. It is very easy to make mistakes. It can be very hard to make sense of, or verify independently, the work of others. And so, through necessity or laziness, work is often judged superficially on the names (of people, institutions, journals) or the appeal of the 'story' told.

We must find ways to push back these subjective incursions.

Technology is part of the way forward. The internet is an infant, yet already powerfully connects people and ideas.

Open Science Lite

Open Science in basic form requires two things: (i) clear and complete presentation of data and methods, and (ii) for the authors to care genuinely about the correctness of their work, and to act in response to any mistakes or problems that arise, before and after publication.

To do Open Science at its most basic is just to respect long-established and well-understood principles and practices.

Web pages are a powerful way of maintaining a record of data, methods, and the thinking upon which decisions are based, and for presenting these in a more accessible way than the peer-reviewed literature generally allows. This web based approach allows for contributions and feedback during and after the 'doing' of the science.

One cornerstone of science is replication, a topic that can bear some discussion. It does not matter where or when or by whom an experiment or analysis is performed: if it demonstrates something that is objective truth, the same answers will arise in the hands of any competent practitioner, anywhere, anytime.

Replication is more complex than it might at first appear; the concept spans a continuum, exemplified at one extreme by the mechanical processing of a recipe, and at the other by total reconsideration and reconstruction of the means by which to test the claim. Recipe-style replication is important for fault finding, but has limited weight in supporting claims (it is necessary, but not sufficient). Conversely, the replication-by-new-analysis might not mean so much when it fails, but acts as powerful support when it succeeds.

To practice Open Science is to embrace critical analysis of your work. This allows for fault finding in the first instance, and enables deeper understanding of conclusions in the longer term.

Open Science - a Stronger Form

OzEA is an experiment in a stronger form of Open Science. It is one thing do our science live on the internet, the work open to scrutiny and comment. It is a further step to present the model we do (The Stories, Data, Analysis, Models, Literature, Discussions). These levels are foundational to the stronger form of Open Science that we seek to facilitate and lead; this being to build a place where a virtual community can constructively work on answering the questions we set ourselves.

Most of us have preconceptions as to the likely outcomes of this work; however, even within the OzEA core team these preconceptions pretty much span the rational spectrum on touchstone topics. We do not renounce our preconceptions, we just leave them at the door. Careful treatment of data, clear-sighted analysis, robust and coherent models -- these things will transparently dictate the claims and conclusions that arise here. A diversity of preconceptions helps to prevent any particular view of what should be the answer subtly biasing the work. In this way the results that develop here can be demonstrably correct, and thus lay claim to the authority of scientific objectivity.

A clear goal here is to develop the interactivity of the web beyond that of the forums and blogs with which many are familiar (and familiar also with their failings and limitations). Hence the rules of etiquette and the separation of data from analysis from our stories, the models, literature, and discussions (i.e. the menu bar). These divisions are tentative, but the logic underpinning them is not. By demarcating the introduction of datasets to the datasets area, by focusing the consideration of literature into its own area, and by maintaining these divisions we achieve two important things; first, we have a defined work flow that can be iterated in waves of development (you may need to observe this in action to properly understand our method), and second, problems and controversies are contained into the areas where resolution most likely lies, and thus need not cripple the entire flow of work.

We are developing methods and standards for high-quality Open Science. We seek to develop a community here working on the renewable penetration problem: a diverse, smart, friendly and open community. We offer a model and a place to engage. Join with us.


[Jump to bottom]

DISCUSSION: (on open science and its application here)

2

OzEA_TOSC0002

John Morgan
Subject: Curating open science
Date: 2010-05-26 (at 14:37:20)


This is a good vision. I particularly like the high level decision to separate data from analysis etc (a bit MVC). This is bound to be necessary as contributions grow.

"Open" means many different things to many people. At one end of the spectrum I could read the above and conclude that you - the oz-energy team - simply mean to be open about the science you do. At the other end, it might mean completely open collaborative work among any and all contributors. I assume you are considering an activity somewhere in between.

The in between I would hope for is for openness to be tempered by a strong curatorial responsibility. The subject matter is inherently multidisciplinary, with much of the necessary expertise to be found outside traditional academic environments, in engineering companies, project developers, utilities, community groups, etc. Not all valuable contributors may have a strong sense of scientific rigor, be be able to divorce analysis from interests. So I see the need for a sense of curation. The curator should act as a point of quality control, as well as a narrator, to be able to say where we have come from, where we are, and where we are going in terms of the research.

If original data and analysis are sought, this comment format won't do for long. The ability to upload datasets, graphs, code, longer form and formatted text, etc. will be necessary. I assume you're thinking about this. In my experience working in a strongly R&D based and very multidisciplinary company, we have found internal open publication in the form of a corporate wiki to be very effective. This allows individual scientists and engineers to publish their work at their own discretion, with accompanying data, and open their work to review from all, with visible commentary and additional data from others. It works very well for us, and may also be the sort of mechanism that would suit your aims.

Something else that might come up - I suspect a lot of the data you would like is commercial in confidence - eg. wind farm output, or project costs. Or maybe licensed data sets. That won't reconcile very well with open science. Can you open up paid-for data here? How do you do collaborative work with closed data sets? Can you access commercial data? What if such data falls off the back of a truck? Maybe giving some thought to anonymous submission mechanisms, anonymous identities, and the legalities and ethics of such may be prudent.

3

OzEA_TOSC0003

Francis
Subject: Re: Curating open science
Date: 2010-05-27 (at 09:04:41)


Thanks John, you make several important points.

We seek to be as open as we can be; just what this means will come out in the doing. And when you hope for openness to be tempered by a strong curatorial responsibility, I can hardly agree with you more. If someone is not responsible for each and every part, then mistakes, big and stupid, can all the more easily burry themselves into the workings. Unless otherwise stated it is me who is responsible, and Barry also. You may have noted that most pages are initialled at the very bottom left.

Upload functionality is indeed on our agenda; for the time being, any such issues can be resolved via email (contact us). Note that basic markup (, etc) can be used in comments, although we don't want to encourage excessive use. The general forum is now the place for discussing site site functionality.

Data is generally presumed to be open enough to post until and unless we know otherwise. We don't want to disrespect real IP, but we are not in the habit of looking for reasons not to post data. In the case of 'fair use' we may be able to post examples and summary statistics of data - always feel free to email the responsible person to ask for clarification.

5

OzEA_TOSC0005

Barry Brook
Subject: Peer review
Date: 2010-06-04 (at 21:21:55)


This comment follows from a discussion thread on BNC.

I submit that the Open Science model under which oz-energy-analysis.org operates is potentially* superior to peer reviewed technical journals, and indeed it's a model that's been discussed as a desirable option by working scientists for a number of years. The PLoS journals have made some attempt to move to this format, and I think you will find that, within a decade or two, it may well be the predominant model.

*Peer-reviewed journals could be superior if the best/most appropriate people are doing the peer review, and the Open Science method is not getting that sort of treatment. In practice, I think it will be rare that Open Science, done properly would not get better (and more diverse and incisive) review than that which could be offered by a few anonymous reviewers who only have a chance to comment on the final version, rather than on the process of doing (except to recommend rejection if they don't like the methods).

7

OzEA_TOSC0007

Francis
Subject: Pointed comment on Peer Review
Date: 2010-06-19 (at 17:28:30)


My words were not so eloquent, and so I defer to these:

Richard Horton, editor of the British medical journal The Lancet, has said that:

The mistake, of course, is to have thought that peer review was any more than a crude means of discovering the acceptability - not the validity - of a new finding. Editors and scientists alike insist on the pivotal importance of peer review. We portray peer review to the public as a quasi-sacred process that helps to make science our most objective truth teller. But we know that the system of peer review is biased, unjust, unaccountable, incomplete, easily fixed, often insulting, usually ignorant, occasionally foolish, and frequently wrong.

http://en.wikipedia.org/wiki/Peer_review

[Show Full Lists]


Post Comment:

A name or alias, email and concise subject are required. Your email will not be abused.
Comments are required to be polite and on topic (commenting etiquette)

Name:*
Email:*
Website:
Subject:*
What is the longhand for 'Oz'? :*


fc, Jan 2012 -- [started as: fc & bwb - May 2010]