Planet GridPP

Software Sustainability Institute Blog | Usability for sustainability

2012-02-03T10:18:14+00:00

I spent a couple of days this week at the wrap-up workshop for the JISC Usability Programme. Here in Edinburgh we've just completed a project under this programme to enhance a visual workbench tool for data-intensive research, and the meeting brought us together with a dozen or so similar projects to swap war stories and thoughts.

I have to confess to being fairly new to many of the ideas, even though they've been around for a while (just ask Russell Beale of the University of Birmingham!). However, our project had excellent support from Mike Jackson here at the Institute who has a background in HCI, and a combination of his expertise in heuristic evaluation and the Institute's own software evaluation guide meant that the usability perspective really drove the project development agenda. You can follow the project's progress on the blog over at SourceForge.

NGS | One down two to go

2012-02-02T11:33:59+00:00

Yesterday saw the first presentation in our short seminar series concentrating on the recent developments in the UK for accessing and managing grid resources.

I'm pleased (and relieved!) to say that it went well with Mike Jones from the University of Manchester giving a presentation on "Shibboleth Access to Resources on the NGS". We had 28 individuals join us on Evo from all over the world including Russia, Italy, USA and Switzerland. It was good to see that our seminar was of interest to people internationally as well.

The next seminar will take place on Wednesday 8th Feb at 10.30am (GMT) and will be looking at the Certificate Wizard which makes it easier for users to manage their certificates. If you would like to take part in the seminar either by Access Grid or Evo then please see the event listing on our website. You can also RSVP on our Facebook event page.

The seminars have been recorded and it is our aim to have these recordings available on the NGS website at the end of the seminar series.

Software Sustainability Institute Blog | Can you code better than your friends? Find out at Dev8D.

2012-01-31T16:07:10+00:00

Last year, I went to Dev8D and was extremely impressed by the enthusiasm shown by the three hundred developers who attended. This year, I'm happy to say that the Institute has been invited to present a workshop about sustainability, but there was a proviso: it has to be exciting. Now there's a challenge.

Developing sustainable software is mainly about good software engineering, and good software engineering is like regular exercise, or eating your five, daily portions of fruit and veg. It pays off in the long run, but it's not always edge-of-the-seat kind of stuff. Fortunately, Mike Jackson came to the rescue with an excellent idea: competitive coding.

The workshop will take place on Wednesday 15 February in room 3E at 10.00-12.00 - see the programme for details.

At the workshop, we're going to pair people up and get them to assess each other's software - with a focus on the things that make software sustainable. We'll start off with some of the simple stuff, like can you find the project's website armed only with the name of the software? And then we will build up to more complex questions about how the software is written: is it readable, installable, designed well? There will also be an open session at the start of the workshop, where we will discuss what makes good code good.

It's easy to listen to us talk about good software engineering, but it's quite a different experience to have a friend look at your code and tell you what they think. And it's not just about learning a few home truths, it's about a fresh insight which can lead to new and fruitful changes and improvements.

NGS | Interested in accessing and managing grid resources?

2012-01-26T16:00:41+00:00

If so then read on!

The NGS is hosting a short but sweet seminar series starting next Wednesday (1st Feb). There will be 3 seminars over the 3 weeks each lasting approximately 30 minutes and the best thing about them is that you can join in no matter where you are - all you need is the internet!

We wanted to make the seminars as open to everyone as we possibly could and, after some deliberation, we decided to use the Evo technology. This is free for everyone to use - all you have to do is to register and I recommend doing this at least the day before. This isn't anything to do with Evo's registration process more that it took several hours for my university email system to allow my confirmation email through...

So what are the topics that we will be discussing?

1st February - Shibboleth Access to Resources on the NGS – Mike Jones, NGS, University of Manchester
This talk will demonstrate how it is possible to access and use NGS resources using institutional login credentials (via the UK Access Management Federation). It will describe how the UK's two main e-Science authentication systems are combined to form an easy to use yet robust identity management environment. It will discuss how this mechanism links together with system, project and Virtual Organisation (VO) registration procedures.

8th February - Certificate Management in the UK - John Kewley, NGS, STFC Daresbury Laboratory
The NGS helpdesk receives many tickets relating to certificates (and certificate renewal in particular): largely due to browser incompatibilities. In order to tackle this problem, the NGS has devised CertWizard which is a browser-independent certificate tool. The presentation will give an introduction to the UK e-Science CA, which has issued over 30,000 certificates, and its associated software and interfaces, including CertWizard.
It will show how modernisations are being made at various stages of the certificate lifecycle, making it easier than ever for users to manage their e-Science Certificate.

15th February - Moonshot - next generation federated identity - Josh Howlett, JANET
Federated identity yields significant benefits for users and services by increasing the usability of services, reducing identity management costs and improving regulatory compliance.
A number of different technical strategies for federating identity have emerged during the past decade, with differing levels of success. These technologies address different types of use case, resulting in significant complexity for both users, services and trust infrastructure providers.
This complexity impedes the adoption of services and increasing operational costs. Moreover, there are many use cases where these technologies do not provide a solution.
Project Moonshot is an ambitious Janet-led initiative, building on existing deployed technologies, that aim to develop a single unified and standardised approach that satisfies all of the authentication and authorisation requirements of the education & research community. Much of the technology has now been implemented, and is now being tested within the Janet Moonshot Technology Pilot.
This presentation will provide an overview of some of the motivating use cases for Moonshot and an overview of the technology and the implementation.

Full details of how to join the seminars are available on the NGS website event page but if you have any queries then please contact the helpdesk and we will do our utmost to help you join in.

NGS | It's that time of year again...

2012-01-19T16:11:13+00:00

My inbox seems to be full of emails regarding conference calls for papers, early bird registrations, conference deadlines etc. Yes it's conference preparation season and its in full swing!

I received confirmation today that I'll be giving a paper at the forthcoming EGI Community Forum on our champions networks. I'll be talking about both our Campus and Community champion networks and how we work with each other to promote e-infrastructure in the UK. Several other NGS staff have also had papers accepted on topics including "Linking Authenticating and Authorising Infrastructures in the UK NGI (SARoNGS)" (Mike Jones) and "Tweaking the Certificate Lifecycle for the UK eScience CA" (John Kewley).

Also in my inbox this week was an announcement from the Software Sustainability Institute (SSI) announcing that registration for their Collaboration Workshop 2012 (CW) is now open. This is on of my favourite events as, unlike most conferences, you don't sit passively listening. The CW consists of breakout groups where you discuss topics submitted by the attendees and there's always one of interest to me in every session. After the discussion a member of the break out group volunteers to report back to the CW as a whole. This means that you get to hear what all the other break out groups were talking about and you can still feedback on their outcomes as well.

It's a really lively meeting and you leave after 2 days feeling tired but feeling that you've achieved something worthwhile! It's also a great place for networking with new people as there are researchers from a wide variety of research areas, IT people, community support people and people like myself who represent national initiatives. To see some of the topics already suggested for discussion visit the event website.

Software Sustainability Institute Blog | How easy is it to teach software skills?

2012-01-19T11:59:59+00:00

Regular readers of this blog will know that the Software Sustainability Institute has been collaborating with the Software Carpentry initiative to develop and deliver courses. Greg Wilson from Software Carpentry has set up a Peer2Peer University course on "How to Teach Webcraft and Programming to Free Range Students". One of the things that the SSI has become aware of as it has undertaken projects is that the experiences and skills in programming of researchers varies greatly, even within an research domain or group.

As part of the first exercise, members of the course have been considering the recommendations published in 2007 by the US Department of Education’s Institute of Education Sciences in a 60-page report: Organizing Instruction and Study to Improve Student Learning. The seven recommendations are summarised below, but the full report is worth a read as it contains a great deal of evidence to back up the validity of the recommendations and other claims.

Space learning over time. Arrange to review key elements of course content after a delay of several weeks to several months after initial presentation.
Interleave worked example solutions with problem-solving exercises. Have students alternate between reading already worked solutions and trying to solve problems on their own.
Combine graphics with verbal descriptions. Combine graphical presentations (e.g., graphs, figures) that illustrate key processes and procedures with verbal descriptions.
Connect and integrate abstract and concrete representations of concepts. Connect and integrate abstract representations of a concept with concrete representations of the same concept.
Use quizzing to promote learning. Use quizzing with active retrieval of information at all phases of the learning process to exploit the ability of retrieval directly to facilitate long-lasting memory traces.
Help students allocate study time efficiently. Assist students in identifying what material they know well, and what needs further study, by teaching children how to judge what they have learned.
Ask deep explanatory questions. Use instructional prompts that encourage students to pose and answer “deep-level” questions on course material. These questions enable students to respond with explanations and supports deep understanding of taught material.

Many of these recommendations are aimed at more traditional notions of students - the SSI is training those who hae already undertaken university degrees: typically PhD students and early-career researchers, though also all the way up to estbalished professsors.

Northgrid-tech | DPM database file systems synchronization

2012-01-15T08:51:12+00:00

The synchronisation of the DPM database with the data servers file systems has been a long standing issue. Last week we had a crash that made more imperative to check all the files and I eventually wrote a bash script that makes use of the GridPP DPM admin tools. I don't think this should be the final version but I'm quicker with bash than with python and therefore I started with that. Hopefully later in the year I'll have more time to write a cleaner version in python that can be inserted in the admin tools based on this one. It does the following:

1) Create a list of files that are in the DB but not on disk
2) Create a list of files that are on disk but not in the DB
3) Create a list of SURLs from the list of files in the DB but not on disk to declare lost (this is mostly for atlas but could be used by LFC administrators for other VOs)
4) If not in dry run mode proceed to delete the orphan files and the orphan entries in the DB.
5) Print stats of how many files were in either list.

Although I put few protections this script should be run with care and unless in dry run mode shouldn't be run automatically AT ALL. However in dry run mode it will tell you how many files are lost and it is a good metric to monitor regularly as well as when there is a big crash.

If you want to run it, it has to run on the data servers where there is access to the file system. As it is now it requires a modified version of /opt/lcg/etc/DPMINFO that point to the head node rather than localhost because one of the admin tools used does a direct mysql query. For the same reason it also requires dpminfo user to have mysql select privileges from the data servers. This is the part that really could benefit from a rewriting in python and perhaps a proper API use as the other tool does. I also had to heavily parse the output of the tools which weren't created exactly for this purpose and this could also be avoided in a python script. There are no options but all the variables that could be options to customize the script with your local settings (head node, fs mount point, dry_run) are easily found at the top.

To create the lists it takes really little time no more than 3 minutes on my system but it depends mostly on how busy is your head node.

If you want to do a cleanup instead it is proportional to how many files have been lost and can take several hours since it does one DB operation per file. The time to delete the orphan files also depends on how many and how big they are but should take less than DB cleanup.

The script is here: http://www.sysadmin.hep.ac.uk/svn/fabric-management/dpm/dpm-synchronise-disk-db.sh

NGS | Just incase you missed it....

2012-01-12T11:10:45+00:00

A new edition of the quarterly NGS newsletter was released in December so if you missed it in the pre-Christmas rush, now is a chance to catch up!

This edition featured articles on -

the adoption of Globus Online by the NGS
NGS involvement in the EGI Federated Cloud Task Force and the benefits for NGS users
NGS user case study - Scalable Road Traffic Monitoring using Grid Computing
...and more!

I am always looking for new articles or suggestions for articles for the newsletter so if you have anything you would like to see in the next edition (March) then please let me know. The next edition will coincide with conference season so copies of the newsletter will be distributed at the forthcoming SSI Collaboration Workshop and the EGI Community Forum.

NGS | Happy new year!

2012-01-05T11:31:19+00:00

I hope you all had a good Christmas and New Year break.

It's back to work and planning for the future here at the NGS with several future events on my to do list.

At the end of last year I finished off the last of our user case studies which highlight how our users have used NGS resources and the advantages it has brought them. The full set of case studies numbers 29 with the latest arrivals listed below -

Using the NGS to run a computer tournament on social learning strategies - Luke Rendell, University of St Andrews

Accelerating the Processing of Large Corpora: Using Grid Computing Technologies for Lemmatizing 176 Million Words Arabic Internet Corpus - Majdi Sawalha, University of Leeds

Computer Simulations of Biological Molecules at the Atomic Level - Sarah Harris, University of Leeds

The next set of case studies are at the planning stage but these will take a slightly different direction. The next set of case studies will look at how the NGS is working with large national and international projects to fulfill their objectives.

Also on the horizon is Easter conference season with several events coming up including the EGI Community Forum which will take place in Munich in March. Several NGS staff have submitted abstracts to this event highlighting work we have carried out in various areas including champion networks, authorisation and authentication.

The week before Munich is the Software Sustainability Institute Collaboration Workshop which you may remember from previous years. This year the event will be held in Oxford and the NGS is involved in several ways including holding a session for our SeIUCCR Community Champions. Watch this space for more information!

Apart from attending other organisation's events I have one of our own to organise. Following the success of last years SeIUCCR e-infrastructure summer school, we will be holding another summer school this year. After the deluge of applications we had last year, I am planning to advertise earlier this year to give a little more time to go through all the applications! Again keep an eye on the NGS website and our mailing list for more information.

Software Sustainability Institute Blog | Scala: fewer lines of code and better pay?

2011-12-22T13:45:43+00:00

By Joanna Leng, independent computational scientist.

I was recently thinking about Java accreditation, but a friend suggested that I should learn Scala instead. I did a quick web search and found that Scala was more compact than Java (it required fewer lines of code to produce the same outcome) and that Scala programmers tend to be paid more than Java programmers. So when I saw a talk called Scala kickstart I decided to give it a go.

The talk was given by Jan Machacek from Cake Solutions (the slides are also available).

So what is Scala?

It is a fusion language that combines object oriented with functional programming (it is not a purely functional programming language though).
It is statically typed.
It is easy to adopt, because it works with existing Java byte code and produces Java byte code. Scala is normally used in combination with Java either because you are building upon legacy codes or because you need the Java Swing UI.

It also has some cool features:

It can convert text to speech.
The acca libraries which handle parallel programming and multithreading well - better than Java.

The talk went through six elements that you need to be proficient in Scala, and it was clearly and well presented. The official Scala web site also looks to be a great resource and, of course, you can download Scala to try it out.

ScotGrid | Batch system juggling

2011-12-21T22:11:09+00:00

We've been a bit quiet up here recently. This is normally a sign of either nothing interesting happening, or entirely too many interesting things happening. Opinions on that may divide, but I think it's closer to the latter...

One of the recent bits of fun that occurred was with our batch server. This story actually starts a long time ago; about this time last year. At that point, we started to get intermittent memory errors from the Torque server - corrected by ECC - but that's generally a sign that the RAM's about to fail. Given that the batch server is single point of failure for a site, that's not a good thing.

So I spent some time preparing a spare box, and being ready to move the batch system over, in case it failed over the winter break. Which, after all that prep, it didn't, and the errors stopped. On the expectation that the current hardware was nearing end of life, we ordered a new box early this year, and have had it sitting in a machine room for a while.

Unfortunately we didn't get time to have it running a tested batch system until our power supply started to ... well, insert colourful metaphor here, describing the 8 months where we were affected by lack of power.

Power got to stable supply in September, and so to catch up on things. One of the things we got around to was software versions. Whilst we didn't intent to update the Torque version, and managed to avoid it for a bit, the gLite developers eventually managed to sneak the update past us as part of an ordinary gLite update. Strictly, this didn't affect the batch server, just all the CE's, making them incompatible with the previous version of Torque.

Whilst a clever manoeuvre, reminiscent of Odysseus' Pony, it did leave us with a conundrum of either reverting the gLite update, or running forward with it. Neither were options of good character, but running forward did have some actual documentation; hence it was full speed ahead.

Which worked out well enough. The Torque 2.5.7 packages were set to use Munge, so getting that installed and tested as a first step helped it go smoothly. To preserve compatability in file locations, we used /etc/sysconfig/pbs_mom to put the pbs working directories in the same place as previously - meaning we didn't have to reconfigure any other tools.

What didn't go so smoothly was the memory leak in the server.

Which gave it a runtime of around 36 hours between crashes. Actually, not even crashes - we found that the pbs_server process hit either

12/05/2011 10:19:12;0080;PBS_Server;Req;req_reject;Reject reply code=15012(PBS_Server System error: No child processes MSG=could not unmunge credentials), aux=0, type=AlternateUserAuthentication, from tomcat@svr021.gla.scotgrid.ac.uk

10/29/2011 18:11:24;0001;PBS_Server;Svr;PBS_Server;LOG_ERROR::Cannot allocate memory (12) in send_job, fork failed

and then sat around moaning. Had it crashed hard, then the auto-restart would have caught it. Ho, hum, one for the Fast Fail philosophy there.

By this point, my proof reader is pointing out that I started off talking hardware, and now talking software. Punchline is that the new server that we never got a chance to use has a lot more RAM than the old server. Therefore we wanted to move the server from the old hardware to the new, to give it a lot more RAM space. That won't fix the memory leak, but will mitigate the problem a bit.

Conventionally, this would involve draining the cluster, repositioning the CE's and then starting up everything again. Had we done that, this blog post would be over now.

Instead, we did a rolling update. This let us move things over without having to do a full drain. The biggest problem with a full drain is that, while most of the jobs finish within a shorter period of time that then limit, there are always some that take the full duration. This leaves us with an empty cluster, doing nothing, for 24 hours or so, wainting on a couple of jobs to finish.

So, instead, by moving things in small batches, then we can keep most of the nodes working, and thus get more work out of things. Step zero is to disable cfengine, otherwise it tends to try and 'fix' things part way through.

Step one is to drain a CE, which we did over a weekend, and a small number of nodes, which we put offline on the Sunday morning.

Come Monday, I set up and tested basic operations with the new batch server, and then moved the freed up nodes across to it. Once those were tested (which shook out a couple of issues about versioning of some libs), point the CE at the new batch server, and then run a test job though it. (It turns out that Atlas are fast enough to sneak some pilots through a 2 minute window for a test job. However, only a few, so they actually functioned as effective tests, without compromising the site if they failed).

After that, it's time to offline another CE, and then some more nodes, and start moving nodes over when they were empty. In the end I scripted this:

#!/bin/sh

NODE=$1
RUNNING=$(qstat -n -1 | grep $NODE | wc --lines)

if [ "x${RUNNING}" != "x0" ]
then
echo $NODE: Still $RUNNING jobs going, skipping
exit 2
fi

CORES=$(qmgr -c "print node ${NODE}" | grep "np = " | cut -d= -f2)

FROM=svr666
TO=svr999

echo $NODE: Moving to ${TO} with ${CORES} cores

ssh ${TO} "~/addNode.sh ${NODE} ${CORES}"

ssh ${NODE} "service pbs_mom stop"
scp config.mom.svr666 ${NODE}:/var/spool/pbs/mom_priv/config
ssh ${NODE} "service pbs_mom start"

ssh ${FROM} "~/deleteNode.sh ${NODE}"

In theory one can run qmgr remotely, rather than ssh-ing to the batch servers and running a script. In practice, with the different versions of Torque, I couldn't get that to work. Note the automation of the mom config switch as well; and that this script checks that the node is empty.

This reduced the gradual move of nodes to a process of croning the script, and offlining nodes occasionally.

The net result was that we were operating at around 80% capacity for 48 hours, and it was all rather uneventful - in a good way. The final step was to update cfengine config and re-enable it.

One of the plus points of the above script is that it should be simple to adapt to two distinct batch systems; which means if we end up moving away from Torque, we should be able to do that without downtime too.

Software Sustainability Institute Blog | How many funding aggregators do we need?

2011-12-20T15:04:08+00:00

By Simon Hettrick.

Last year, I found myself talking about funding aggregators and whether we could link to them on the Institute’s website. At the time, I thought that it was a little strange: why the plural? Why would anyone need more than one aggregator? At first, there might be an aggregator for UK funding, one for Europe and one for industry, but surely that wouldn’t last for long, because someone would just write an uber-aggregator that combined all three. It turns out that this might have happened.

I’ve just added Research Professional to our list of useful resources. It cites itself as “the world's leading provider of news and funding information for research professionals” and covers “academia to politics, technology to the arts”. The people who run the service are called Research (not a good name if you wish to prevent confusion in the research market, but let’s overlook that problem) and produce a number of publications from Research Europe to Research Caribbean. In fact, they say that provide the “largest database of research funding opportunities available worldwide”.

I took a quick look at the Research Professional website, and it certainly seem comprehensive. After logging in with my university email address, I was presented with a handy interface into which I typed as many random research-related words as I could think of. I was quickly presented with funding opportunities from all round the world. The search can be focussed by selecting specific countries, funding agencies, closing dates and many, many more variables. And a lot of information is presented for each call, such as the closing date, the award type and the frequency with which the call is made.

NGS | The Training Marketplace

2011-12-20T14:25:40+00:00

Claire Devereux from the NGS introduces the EGI Training Marketplace.

The Training Marketplace is a service that allows you to search for or advertise training events and resources throughout the EGI community. These can be events or resources that may be open to absolutely anyone, just those within the EGI community, or they may be specific to a small project or to one country only. The Training Marketplace is the one-stop shop for your training needs as a user and is open and free for both academic and commercial providers to advertise in.

Since May the Training Marketplace has evolved from a simple event and material repository into a interactive site where requirements can be captured, events can be rated, and the tool can be customised and embedded into third-party websites using our gadget generator.

The following types of resources are currently supported:

traditional training events, usually classroom based or workshops where people attend in person. They can also include virtual events running at specific times. The difference between training events and online training is that training events have a set start and end time whereas online training is accessible either permanently or over a longer time scale.

You can choose to display events as a list, on a calendar or on an interactive map.

online training, available via the web. This category covers a wealth of resources, from self-study courses that require users to log in and complete exercises at their own pace leading to a qualification, through to online tutorials that users can tap into as they wish.
training resources, physical resources or services available to the community to assist in training. An example of a training resource is the GILDA Certification Authority. GILDA issues temporary (14 day) personal public key certificates (compliant with the X.509 standard) in order to access the GILDA Testbed for user training.
the requirements area is a place for users to describe their training needs and for training providers to see if there is interest in running courses. For example, a provider may post details of potential offerings and ask those interested to visit their website for further details, to check for viability before running a course.
University courses is the place for higher education institutions to advertise their Masters and Doctorate training opportunities.

After attending an event or taking part in online training users can rate the event and leave their feedback, letting others know the value of their experience much in the same way are people do with online shopping nowadays.

The latest release is a step towards improving the user's experience and looking towards longer term sustainability. It includes the new online training category, much improved search functionality, notifications to authors once entries are published and an improved calendar view. We are now working improve the functionality and appearance of the Training Marketplace gadget, which allows projects or NGIs to embed the EGI Training Marketplace into their own website, pick and choose which parts to include and even skin some elements with their project colours. If you are an NGI or project we would be interested in hearing about your requirements for the gadget, so please contact us.

WLCG RAL Tier 1 | RAL Tier1 – Plans for Christmas & New Year Holiday

2011-12-20T11:21:08+00:00

RAL closes at 3pm on Friday 23rd December and will re-open on Tuesday 3rd January. During this time we plan for services at the RAL Tier1 to remain up. The usual on-call cover will be in place (as per nights and weekends). This cover will be enhanced by daily checks of key systems. Some hardware interventions, such as to swap out faulty disks will also take place over this time.

Furthermore we do not have support on 25/26 December & 1st January for some services we rely on. The impact of any failures around these particular dates may therefore be more extended. Also, over the holiday we have relaxed our expectation that the on-call person will respond within two hours, particularly on the specific dates just mentioned.

During the holiday will check for tickets in the usual manner. However, only service critical issues will be dealt with.

The status of the RAL Tier1 can be seen on the dashboard at:
http://www.gridpp.rl.ac.uk/status/

Gareth Smith

Software Sustainability Institute Blog | What makes good code good? A digital social research view

2011-12-16T10:32:51+00:00

By Mike Jackson.

Last week, we ran a sustainability training workshop for Digital Social Research, where we asked "what makes good code good?". The attendees, who were research programmers and software developers, put together a list of necessary qualities, which we've copied here.

Good code should...

Correct. Code must be correct and it should also be possible to demonstrate that it's correct, e.g. through provision of associated tests or mathematical models of requirements.
Well-designed. Code should be modular with well-defined interfaces, inputs and outputs and with code and data encapsulation. It should be elegant and no more complex than necessary. There should be minimal inter-dependencies, no hidden dependencies and limited platform-specific dependencies. Together, these help ensure that the code is easily understandable by other developers; can promote reuse, so reducing the need to reinvent the wheel in subsequent projects; and ensure software can be configured, adapted and extended easily.
Readable. Code should be commented and indented and use sensible naming. Comments should describe why the code is as it is, since the code itself describes what it does and how it does it. Care should be taken that comments reflect the current code, because code evolves through time.
Appropriate. The languages, technologies and tools should be suitable for the intended application area, and also take into account the skills and knowledge of the current and future developers.
Robust. The code must not break anything and it should fail gracefully. Ideally, it should support configurable logging or other ways to help users and developers identify and diagnose errors. Errors must not be swallowed by the code without a good, and commented, reason.
Efficient. Code must run in a timely way, for the specific applications area.
Available. Software should be available to those who need it! If it's not available, how will anyone be able to use it?
Usable. Software should be usable, buildable, deployable and runnable. Difficult-to-use software can discourage its uptake by users. Software that can't be built, deployed or run is highly unusable!
Copyrighted and licenced. These protect intellectual property and let others know how they can use, modify and redistribute it.
Under revision control. The revision control should be backed up, and supported by sensible commit messages.

NGS | CA stuff

2011-12-15T12:27:41+00:00

Three almost unrelated things, except that they relate to the CA(s), somewhat technical stuff, so bear with me:

If you have renewed your certificate recently and found that it didn't work with VOMS, this is due to a misdesigned "feature" of VOMRS that it insists on registering not just the user name but also the issuer name as part of the account. In the Real World(tm), we can keep the issuer (ie CA) name the same all the time but this does not work with grid middleware, so we have to change issuer name whenever we roll over. The "feature" adds no extra security for grid CAs because the user name will always be unique. There is a workaround that tells the server to ignore the feature, which we thought everyone had been using for years - but this, as Steve Traylen points out, will still cause problems if your certificate expires before you can resign the AUP (surely a rare case?!) - but should otherwise work fine. The best option otherwise seems to be to get your VOMS admin (not VO admin!) to duplicate the account entries, one with each issuer name, the old and the new one. CERN (Steve) has done this with theirs, and we are checking the other ones to see if they have failed to enable skipcacheck. I am both amazed and sorry that we have not discovered (and resolved) this sooner... but Steve is a Wizard(tm) and will fix it... Incidentally, it is not just us, other CAs roll over too - however, many have chosen to extend the lifetime of the existing certificate instead of renaming it - this will then not cause problems with the VOMRS accounts, but it causes problems with server/client synchronisation for HTTPS instead - if the server and client (browser) are not updated in sync (and they never are), some browsers will print obscure error messages and fail to connect (as we know from past experience).
Oh, and another good thing is that the IGTF rules have changed - we can now make the end entity issuing CAs have longer lifetime, so we no longer have to roll over every four years. Hooray!
On the subject of IGTF 1.43 release which came out recently. We're rebuilding the NGS-specific release except we are going back (or forward?) to individual RPMs instead of a single one for the lot - this means we need a couple of dependency RPMs but we should then be able to not mess with the IGTF stuff much and sites can in principle fine tune what they install. We'll have to think about the dependencies carefully.
Related to this (so, er, not entirely unrelated), there is this problem with the IGTF root signing policy file. We're trialing the Least Elegant Workaround(tm) this time, having discussed it at some length, by testing a self-signed version of the SLCS toplevel. This makes it independent of the root in the technical sense of building a verification path and checking signing policy files, so would slot in next to an unmodified IGTF release directly - but the downside is that we now have another self signed certificate that we'd need to establish trust in, and the fact that the policy of the SLCS branch (ie the SARoNGS CA, the CEDA CA, etc.) were supposed to be covered by the root CP. Depending on how good this looks (we're testing it from today), this may appear in 1.43.
Oh yes, and I know I need to get TACAR corrected and updated. This is not trivial (requires writing forms and pgp signatures) so is awaiting a slot where I have some time...

That was three things, for suitably large values of three. If you have any questions, do get in touch...

NGS | Return of the Champions

2011-12-14T12:35:54+00:00

I’ve recently taken over the organisation of the NGS Campus Champions programme here at the NGS and last week I chaired the first meetingof the group.

Our Campus Champions come from a number of universitiesacross the UK and are mainly representatives from IT Services. As for their role and activities on behalf ofthe NGS? Well this was one of the mainthings I wanted to discuss at the meeting!

Thankfully all our Campus Champions agreed with my list ofproposed benefits for them (and us) of being a Champion. In brief the NGS will provide training inusing NGS resources, tools and e-infrastructure; provide access to onlinetraining materials; produce publicity material to help them publicise the NGSand their role as Champion; hold Campus Champion events at relevant events;hold bi-monthly phone calls for dissemination of news and information.

In return the Campus Champions will actively promote the NGSwithin their institution offering advice and advising researchers; display NGSCampus Champions publicity material; pass onto the NGS requirements from theirinstitution and researchers; liaise between researchers, the institution andthe NGS; attend the bi-monthly Campus Champions meeting.

Currently we have 14 institutions with Campus Champions butwe are always looking for more! If youare interested in becoming a Champion then please contact me. Your site doesn’t have to be a member of theNGS for it to have a Champion. Thecurrent list of institutions is:

Canterbury Christchurch
University of Huddersfield
University of Hull
University of Liverpool
University of Manchester
Queen Mary - University of London
University of Oxford
University of Reading
STFC - Daresbury Laboratory
STFC - Rutherford Appleton Laboratory
University of Sheffield
University of Surrey
University of Sussex
University of York

The next meeting of the Campus Champions will take place byphone in February but in the meantime I shall be making some changes to theexisting Campus Champion pages on the NGS website. Watch this space!

Software Sustainability Institute Blog | How BIG is big data?

2011-12-07T16:23:41+00:00

By Chris Morris, STFC.

GenBank now contains 100000000000 base pairs. That's big, in the sense each similarity search visits every record, and there are millions of searches a day. But it's not BIG, in the sense that it fits on one disk, and only takes 200s to transfer at 1Gb/s.

An electron microscope tomography image may contain 8 billion pixels. That's big, in the sense that the noise reduction algorithms take polynomial time in image size. But it fits on a USB stick. The data stream from LHC is big; but most of it is of no significance. But it can be reduced to a one-bit answer, such as "Does the Higgs boson exist?" The Sloan Digital Sky Survey contains several terabytes of data. That's big, since it is all unique data of potential interest. But it would fit on a medium size RAID. A Next Generation Sequencing instrument can record 100TB of data in a day. It will quickly be reduced ten-fold, and then later reduced to a consensus sequence - a whole human genome is 1.5GB.

Whether data is big or not depends on what you want to do with it.

Ask Steve! | Testing questions about testing!

2011-12-05T09:22:33+00:00

Today’s post comes courtesy of Mike Jackson, also from the Software Sustainability Institute. If the Institute was the Dukes of Hazzard television show, with Steve as Bo Duke, then Mike Jackson would surely be Luke Duke. In this post, Mike answers a testing question about testing frameworks in Python.

Software testing is a vital part of software development. It not only allows us to demonstrate that our software satisfies its requirements but to ensure that our software is both correct and robust. Automated software testing provides us with a safety net during development, allowing us to fix bugs, make enhancements and extensions secure in the knowledge that if we break anything then the tests will catch this. After all, there are few things worse than fixing a bug to discover later that, in doing so, we’ve introduced a new one.

Philip Maechling of the Southern California Earthquake Center (SCEC), at USC, recently contacted the Institute with questions about software testing. Philip and his colleagues develop scientific software that outputs computational results into files. These files are typically simple ASCII text files but contain series’ of floating point numbers e.g. time series. Their acceptance testing involves comparing these files to existing reference result files.

Philip posed two questions:

Many unit test frameworks (e.g. JUnit and PyUnit) are focused around instantiating an object, or other software module, within a test class, calling methods on that module, then checking the values returned against expected values. While file comparisons can be done with such frameworks, they are complicated due to the need for floating point compares (which is tricky at the best of times), and differences in header information, or non-significant file contents. So, are you aware of any testing tools designed to support tests that are based on file-based comparisons?
In our file-based comparison tests, we often use the same reference files in multiple tests. In some testing circles, a directory of tests and expected test results are collected into a datastore called an “oracle”. When you want to know the correct results, you look up your test and find the expected result in this oracle. Are you aware of any software unit or acceptance testing tools that support the idea of a test oracle? The concept is simple, and we have implemented a couple of our own oracle datastores, but we seem to re-invent this each project. If there is a standard solution, I am interested in trying it out.

Question 2 is a generalisation of 1, using a set of reference files across multiple tests. As Philip comments, these reference files can be termed an “oracle”. More generally, “oracle” can be used to refer to anything which validates the outputs of a test i.e. checks that the outputs of the software during the test against the expected outputs. So, for example, in a PyUnit test that compares the outputs of a function, for some specific inputs, to some hard-coded values, the comparison code hard-coded values serve as the oracle. If a developer tests a GUI and assesses the correctness of its behaviour then they are serving as the oracle. Douglas Hoffman’s paper A Taxonomy for Test Oracles from Quality Week, 1998, gives an overview and taxonomy of oracles.

For question 1, an internet search did not reveal any Python frameworks that explicitly support tests that involve comparing floating point data files for equality. Even if a framework were available, there would still be work required by the developer to customise it towards the structure and content of their specific files. Two frameworks which adopt such a solution and provide something close to Philip’s requirements are Cram and TextTest. Cram is a framework for testing command-line applications. It runs commands and compares their outputs to expected outputs. The outputs are compared using pattern matching and regular expressions. TextTest is similar but also has support for GUI testing. Outputs are compared directly, but filters are provided to handle run dependant content and floating point differences outwith user-defined tolerances.

One can envisage at least two general approaches to comparing output files of floating point values to reference files. The first is to:

Write a convertor that can be used to convert the output file data format into a simpler format containing just the floating point data.
Write a validator that takes in two floating point data sets and compares these, applying rounding or allowing for equality within defined tolerances.
Write each test to load the expected results from the reference files, the actual results from the output files, apply the convertor to both sets of results, then use the validator compare the two.

The second is to:

Manually convert the reference files into template files. Regular expressions can be used to both handle parts of the files that might vary across test runs (e.g. headers) as well as for specifying expected floating point values.
Write a validator which compares an output file to a reference file, applying the regular expressions in the reference file to assess whether the output file matches.
Write each test to apply the validator, comparing the output files to the reference files.

Personally, I prefer the former solution as it avoids messing around with regular expressions.

For either solution, there are a number of Python libraries that can be used to construct a possible solution. These include:

PyUnit, Python’s unit test library. This has test assertion commands (assertAlmostEqual and assertNotAlmostEqual) for comparing floating point values to a specific number of decimal places or within a specific tolerance.
Python difflib library. This provides functions to compare two files and return the lines for which they differ. This is similar to the output from CVS and SVN “diff” commands. Cram uses difflib.
Python re regular expression library. Cram uses re.
Python filecmp file comparison library.
An introduction to writing regular expressions for floating point numbers.
TextTest (source code) and Cram (source code) are both open source products and it might be possible to reuse their functionality for comparing script files.
Hamcrest library for building “matchers” which are useful for expressing custom comparisons. It has been ported to many languages including Python.

Mike

Software Sustainability Institute Blog | I am not a light bulb!

2011-12-02T14:56:34+00:00

By Simon Hettrick.

Most people turn apoplectic when faced with someone who “thinks outside the box” or attempts to harvest “low hanging fruit”. And rightfully so. We’ve learned to vilify management speak, because it’s wasteful and verbose, but what about its visual equivalent? It’s time that we start saying “NO!” to meaningless images.

The world of software is a grim place if you need an image for a website. This is down to a fundamental problem: you can’t see software. This leads a lot of people to think “you can see computers!”. But there’s only so many times that you can use that data-centre image - with its banks of cold, emotionless circuitry – before things start to get depressing. And it is this tortuous path that causes some people to embrace stock photography with an incredible level of enthusiasm.

There’s nothing wrong with using stock images. It's impossible not to (unless you have your own photography department). I just advise some caution on the images you choose. Take the image on this page, with it’s clever subtext of being the illuminated one amongst dowdy colleagues. It’s is a good image, but it’s also completely generic. Anyone could find a concept in their business that this image could represent, so it will end up being used by everyone from management consultancies to electricians, and everyone – absolutely everyone – in between. This genericide is infectious: if you use a generic image, you will add nothing but blandness to your publicity.

Software Sustainability Institute Blog | How to successfully attract equity investment

2011-12-02T09:32:14+00:00

By Anthony Clarke, Director, Angel Capital Group and member of ACCESS-ICT.

Out of the vast range of R&D projects supported by the EU, only a small number successfully attract external finance to commercialise their results and capitalise on the original EU investment. This means that many potentially very exciting, new technologies never get to the market place - and never get to the equity community. On behalf of the European Investment community, I am delighted to present a new guide that will enable entrepreneurs and research teams to become investment ready.

A key issue for research projects is how to become investment ready, and how to understand investor requirements. Do your R&D results offer the potential for creating a scalable business? Does your research team have the capacity to run a growth business focused on marketing innovative products and services? Is your IP protected and its ownership clear? Do you have a strategy for approaching the investment community and becoming attractive to investors?

It’s a tough and challenging road for entrepreneurs and research teams to commercialise ICT R&D results, but I feel sure that the European investment community will welcome the opportunity to see more innovative, investment-ready propositions coming from the EU R&D framework programmes. I strongly recommend you use the guide and the online tools to increase your odds of successfully attracting equity investment.
I would like to thank all of the private-sector investors and expert organisations that have kindly offered their experience and expertise to provide key content for this guidebook.

I wish you all success with bringing your innovations to market and accessing relevant investment.

Download the Access to Finance guide.

Northgrid-tech | DPM upgrade 1.7.4 -> 1.8.2 (glite 3.2)

2011-12-01T11:21:48+00:00

Last week I upgraded our DPM installation. It was a major change because I upgraded not only the DPM version but also the hardware and the backend mysql version.

I didn't take any measures this time before and after. I knew that becoming an alpha site in atlas was taking its toll on the old hardware and many of the timeouts were from gridftp but there had been a reappearance of the mysql ones I talked about in previous posts at the level that even restarting the service was hard.

[ ~]# service mysqld restart
Timeout error occurred trying to stop MySQL Daemon.
Stopping MySQL: [FAILED]
Timeout error occurred trying to start MySQL Daemon.

So I decided that the situation had become unsustainable and it was time to move to better hardware and software versions.

* Hardware: 2 cpu, 4GB mem, 2x250 GB raid1 -> 4 cores (HT on = 8 job slots), 24GB mem, 2x2TB raid1

There is no why here it was ok when we had limited access but the recent load was really too much for the old machine even with all the tuning. Suspected bad blocks on disks could be possible but no red leds nor hardware errors were reported by the machine.

* Mysql: 5.0.77 -> 5.5.10

Why mysql 5.5? Because InnoDB is the default engine and they have improved performance and instrumentation. On top of other things that we might actually start to use. A good blog article about the 5 reasons to move is this one: 5 good reasons to upgrade to mysql 5.5.

MySQL 5.5 is not in EPEL yet, but I found this CentOS community site that has the rpms and the instructions to install them.

After the installation I've also optimized the database partially with what I had already done in July, partly running a handy script mysqltuner.pl. This last one helps with variable you might not even know and even if you know them it tells you if they are too small. You need to be patient and let pass few hours before run it again.

* DPM: 1.7.4 -> 1.8.2

Why DPM 1.8.2 from glite 3.2? I would have gone for the UMD release or even the EMI one but then glite 3.2 was moved to production earlier than those and since I waited for this release since at least April I didn't think about it twice when I saw the escape route. It was really good timing too as it happened when I really couldn't postpone an upgrade anymore. You can find more info in the release notes. Among other reasons to upgrade: srmv2.2 in 1.7.4 has a memory leak which wasn't noticeable until the load was contained but for us exploded in October and is the reason I had to restart it every two days in the past few weeks.

Below the steps I took to reinstall the head node

On the old head node

* Set the site in downtime, drain the queues and kill all the remaining jobs.

* Turn off all the dpm and bdii services on the old head node

* Make a dump of the current database for backup

mysqldump -C -Q -u root -p -B dpm_db cns_db > dpm.sql-20111125.gz

* Download dpm-drop-requests-tables.sql supplied by Jean Philippe last July

wget http://www.sysadmin.hep.ac.uk/svn/fabric-management/dpm/dpm-drop-requests-tables.sql

* Drop the requests tables. This step is really useful to avoid painful reload times as I said in this other post about DPM optimization and because it drastically reduces the size of ibdata1 when you reload which has also benefits (my ibdata1 was reduced from 26GB to 1.7GB). Still you need to plan because it might take few hours depending on the system. On my old hardware it took around 7 hours.

mysql -p < dpm-drop-requests-tables.sql

* Dump reduced version of the database

mysqldump -C -Q -u root -p -B dpm_db cns_db > dpm.sql-20111125-v2.gz

* Copy both to a WEB server where they can be downloaded from in a later stage.

* Update the local repository for DPM head node and DPM disk servers. Since it is still glite I just had to rsync the latest mirror to the static area.

On the new head node

* Install the new machines with a DPM head node profile. This was again easy since it is still glite no changes were required in cfengine.

* Most of the following is not standard and I put it in a script. If you have problems with users IDs created by avahi packages you can uninstall them with yum removing all the dependencies and let them be reinstalled by the bdii dependency chain. It should work also uninstalling them with rpm -e --nodeps. This leaves redhat-lsb (which is what the bdii depends on) untouched but I haven't tried this last method. Here are the commands I executed:

# Get the dpm DB file
rm -rf dpm.sql-20111125-v2.gz*
wget http://ks.tier2.hep.manchester.ac.uk/T2/tmp/dpm.sql-20111125-v2.gz

# Install mysql5.5
rpm -Uvh http://repo.webtatic.com/yum/centos/5/latest.rpm
yum -y remove libmysqlclient5 mysql mysql-*
yum -y clean all
yum -y install mysql55 mysql55-server libmysqlclient5 --enablerepo=webtatic
service mysql stop
rm -rf /var/lib/mysql/*
# Get the local my.cnf
cfagent -vq
service mysqld start

# Install the DPM rpms
yum -y remove cups avahi avahi-compat-libdns_sd avahi-glib
yum -y install glite-SE_dpm_mysql lcg-CA

# Modify sql scripts for mysql5.5
cd
/opt/lcg/share/DPM/ for a in create_dp*.sql; do sed -i.old 's/TYPE/ENGINE/g' $a;done
grep ENGINE *

# Run YAIM and upload old DB
cd
/opt/glite/yaim/bin/yaim -c -s /opt/glite/yaim/etc/site-info.def -n glite-SE_dpm_mysql

mysql -u root -p -C < /root/dpm.sql-20111125-v2.gz

# NECESSARY FOR THE FINAL UPDATES
/opt/glite/yaim/bin/yaim -c -s /opt/glite/yaim/etc/site-info.def -n glite-SE_dpm_mysql

* You will need to install the dpm-contrib-admintool rm because it is not in the glite repository it might be in the EMI one. Last time I heard it made it to ETICS. If you can't find it there's still the sysadmin repo version and related notes on the GridPP wiki (Sam or Wahid welcome to leave an update on this one).

* To upgrade the disk servers I just updated the repository, upgraded the rpms and rerun yaim.

Software Sustainability Institute Blog | Celery - not to be sniffed at!

2011-11-30T09:00:00+00:00

By Mike Jackson.

As part of our work with the particle physicists of the MAUS project, I was tasked with extending their data analysis software to use a distributed task queue. The intent is that data analysis tasks be farmed off to worker nodes to be executed in parallel. MAUS recommended using Celery - an asynchronous task queue based on distributed message passing and implemented in Python. I set about my task with some trepidation, and dreading battles with Linux RPMs and building open-source projects from their source code (which always befall me when not using Java software), not to mention the joys of configuring distributed computing software. However, I found myself very pleasantly suprised!

Celery allows jobs, or tasks, to be executed by one or more Celery workers. These Celery workers run on multiple servers. Each Celery worker can also be configured to execute one or more tasks concurrently. A Celery client creates and submits a task which is dispatched to an available worker node for processing. The client can submit their request synchronously or asynchronously - Celery provides support for querying the status of each submitted task.

Software Sustainability Institute Blog | Is Open Government Data ROARing into life?

2011-11-29T13:11:58+00:00

By Ramine Tinati, Web and Internet Science, University of Southampton

Software sustainability is, more often than not, seen to be limited to the development and maintenance of platforms, software systems and applications. But sustainability can encompass much more, especially when dealing with new and developing activities on the Web. In the WAIS group, we are using a new platform, called ROAR, to examine the sustainability of Open Government Data. We have shown that some governments are champions of open data, while others show less of an interest.

Over recent years, the drive for Open Data - specifically, Open Government Data (OGD) - has become an increasingly popular Web activity, driven by Governments, civic organisations, businesses, developers, researchers and citizens. The concept driving the OGD initiative is a fairly simple in theory, but complex in practice: governments need to release their data for public consumption. As with any large-scale initiative, there are a number of interconnected social, economic, political, financial, legal, and technological barriers that must be overcome to enable sustainable OGD communities. A battle of licensing, policy, cultural change, and economic value stand in the way of progress. Technologically, the push for OGD has led to a number of developments in new and existing technologies, including the Semantic Web and Linked Data, which aim to provide the newly freed data with a linked data infrastructure. There has also been the development of new software platforms, such as CKAN and Socrata, which house and manage the data and provide simple interfaces for data access.

How can sustainability be measured? If OGD is examined as a network of stakeholders, its sustainability could be investigated through qualitative means: through interviews, observations, etc. Quantitative methods can also be used, such as monitoring and tracking the publicly available government data. This provides a truthful, unbiased perspective on the overall success of the network. Ultimately, the amount of available data is a reflection of the efforts of the stakeholders involved.

I use ROAR (Registry of Open Access Repositories) to examine the sustainability of OGD initiatives. ROAR is a platform that enables the monitoring of dataset deposits within repositories and data catalogues, providing a way to examine the frequency and size of deposits over a period of time. Based on these measurements, the overall health of the data catalogues can be determined. A healthy catalogue can be classified as one which has a regular frequency of large deposits, thus constantly increasing the size of the available data.

We are currently tracking a number of Open Government Data portals such as data.gov (the US catalogue of open data) and data.gov.uk (the UK equivalent). By examining the health of these portals, we aim to provide a service to encourage transparency on the continuous efforts being made within OGD, raising awareness of the actions of the data publishers. To date, we have been identified a mix of commitment levels towards the publication of government data, with the UK and the US (who were champions of the OGD initiative) demonstrating healthy signs of sustainability, unlike others (naming no names), whose commitment is questionable.

This is just the beginning of an emerging Web activity, and as the number of new OGD communities and portals emerge over the coming months, ROAR will provide the tools required to assess the sustainability of OGD.

NGS | Dinosaurs, DNA and nuclear power

2011-11-28T11:08:46+00:00

Just a normal month or so in the life of the NGS really butwhat do all 3 have in common?

They were all research areas investigated using NGSresources.

The research of William Sellers, Phil Manning and Karl Bateson dinosaur locomotion was featured as a “success story” on the EGIwebsite. They talked about how they usedGrid computing to understand better how dinosaurs moved around and what rolesthey played in their ancient world. Asthere are no similar animals around today to compare to dinosaurs such as a T. Rex,the solution is to create a detailed computer simulation of the animal’sskeleton and muscles.

Not only was their research picked up by EGI but it alsofeatured in iSGTW – fantastic publicity for the researchers and for the NGS!

I’ve been busy putting together some user case studies overthe last few weeks and I’m pleased to say that there are now a few more up onthe website showcasing the large spread of research areas that the NGS facilitates.

First up is Charlie Laughton from the University ofNottingham who has been using the NGS for quite some time now. He used the NGS to investigate the flexibility and folding properties of DNA as understanding how the tightly packed DNA inhuman cells can still be read can, in turn, help to understand how cells switchgenes on and off. At present there is no clear understanding of howthis works. Being able to influence thisin new ways may ultimately help to find new drugs to treat diseases such ascancer, develop new biofuels, and crops that can resist climate change.

Charlie said “…without the compute power and high-throughputprovided by the NGS, we would not have been able to deliver our part of theproject in a timely manner. At a more personal level, it led to one of the mosthighly cited publications I have ever had.”

John Allen from the University of Edinburgh explained howthey use NGS resources to power the GridQTL portal which is used worldwideto study gene expression in a wide range of organisms.

The team’s use of the NGS has greatly increased theproductivity of their users (currently around 400) in the QTL community. Oneexample of this is a GridQTL user at the University of Missouri Columbia. They ran a series of studies on carcass, post-natal growth and reproductivetraits in commercial Angus cattle and found a speed up of from 20 people-weeks,using their old single server system, to 3 people-weeks to capture and analysethe data with GridQTL.

Finally we have nuclear power! Paul Martin from the University ofHuddersfield has been using the NGS to investigate the suitability of Thoria asan alternative form of nuclear fuel. Paul’sresearch is particularly timely as there is increased interest in the use ofthorium dioxide for nuclear power rods not least because of its comparativelyhigh abundance in the earth’s crust and low cost. It is for this reasonthat, although the main fuel for nuclear power reactors is currentlyurania-based, thoria-based fuel is attracting much attention as an alternativehigh performance nuclear fuel.

All of our case studies can be found on the NGS website andwe now have a collection of 26 covering a wide range of research areas. If you are interested in using the NGS casestudies to promote grid resources then please let us know.

NGS | Interact with the NGS

2011-11-28T11:07:20+00:00

I recently updated the NGS presence on Facebook to a new interactive page. Instead of the old-style group page, the NGS now has a new page under the UK NGI banner which reflects our role as the lead in the UK National Grid Initiative along with GridPP.

By “liking” our page you can receive news updates from the NGS all in one place as well as news updates from the Software Sustainability Institute (SSI) and GridPP. Please feel free to invite your friends and colleagues to "like" our page and to share articles with your friends list - help spread the word about e-infrastructure!

GridPP storage news |

2011-11-24T05:56:08+00:00

The best rate to get from ATLAS's SONAR test involving RAL can be assumed to be internal transfers from one Space token at RAL to another space token at RAL. The Sonar plot for large files; (over 1 GB,) for the last six months is:

Averaging this leads to:

Leading to average of 18.4MB/s as the average rate with spikes in 12 hour average to above 80MB/s. (Individual file transmission rates across the network (excluding overhead) have been seen at over 110MB/s. This relates well to the 1Gbps NIC limit on the disk servers in question.

Now we know that of Storm,dCache,DPM and Castor systems within the UK that Castor tends to have the longest interaction overhead for transfers. Overhead for RAL-RAL transfer varies for the last week is between 14 and 196 seconds with an average of 47 seconds and a standard deviation of 24 seconds.

NGS | NGS at SC'11 - round up

2011-11-23T10:42:14+00:00

The UK presence was fairly significant this year at SC'11 with attendance by David Wallom, NGS Technical Director and a significant number of the leaders of research computing centres from around UK universities. This included NGS member sites @ Bristol, Leeds, Oxford, Southampton and EPCC.

In this blog post, David gives us a round up of NGS activities at this major computing event.

Having got off of the very long flight from Heathrow
to Seattle we settled onto the metro to get us downtown. After passable sleep the following morning we headed over to the SC’11 venue – the Washington Convention Centre to collect our badges and then visit the workshop on HPC in Smart Grid, where there was UK interest from the EC FP7 HiPerDNO project being presented by Dr Stef Salvini, OeRC. Following a very productive day where we learnt the state of the art in US Smart grids, how they intend to utilize knowledge developed through the national e-infrastructure for research. We then met up with the EGI team who had an exhibition stand at the conference.

The Monday workshop, Many-Task computing on Grids and Clouds 2011, started off with an interesting keynote from David Abramson (Monash) a long term friend of the NGS through support for their Nimrod tool which is popular with several of our biosciences users. After this there was a panel session which went slightly off topic to talk about exascale more than Many task but it still attracted several questions around the need for exascale, when we are still struggling to get a significant user base onto smaller HPC systems. Overall a good workshop though having the panel first did mean that a number of people didn’t hang around for the rest of the papers. This workshop was operating in a very competitive market with other sessions on cloud and data management which also attracted significant crowds.

The first full day of the conference allowed for the first good look around the exhibition floor alongside several interesting birds of a feather sessions There was also the first of a number of conversations with different groups and vendors, including Microsoft, Mathworks and Adaptive Computing. To give an idea of scale this picture is down one of the main aisles in one of the 5 rooms that were all about this size!!

Pretty impressive stands by a number of people
but the coolest was the multi projection globe on the NOAA stand.

We of course also announced
our activity with Globus Online which created a lot of interest and ended with us having a number of interesting conversations with NSF regarding future collaboration between our national e-infrastructures.

During the meeting the EGI booth was continually visited by a reasonably large number of people, we had the Real Time Monitor showing as normal (having seen a lot of 3D screens this needs to be done in 3d now for next year!). They did though give away a pretty large number of t-shirts as did a lot of stands, so I ended up as the moving poster board around down town Seattle from 6:30-7am every morning on my morning run!

Software Sustainability Institute Blog | GeoTOD-II at All Hands 2011 and Semantic Web 2011

2011-11-21T09:50:39+00:00

The GeoTOD-II project developed a customisable, open-source framework to support linked data interfaces to relational, web service and linked location data resources. The linked data interfaces implement UK Government guidelines on URIs for location data. GeoTOD-II was developed by STFC e-Science under funding by OMII-UK and the Software Sustainability Institute provided consultancy on their use of the OGSA-DAI data management framework.

Two presentations on GeoTOD-II have been given at major UK and international conferences - the UK e-Science All Hands Meeting 2011 (AHM) and the International Semantic Web Conference 2011 (IWSC).

The AHM has been the national forum for UK e-Science development and research e-infrastructure for the last decade. At this year's conference, held from 26-29th September in York, GeoTOD-II's Arif Shaon of STFC presented GeoTOD-II as part of the best papers session. The talk prompted technical questions on whether the framework uses data caching and does its own data integration and also whether GeoTOD-II had engaged with the UK Environment Agency (GeoTOD-II had consulted with the UK Department for Environment, Food and Rural Affairs and provided feedback to the Ordnance Survey). GeoTOD-II won 2nd prize for best paper at AHM!

ISWC is the major international Semantic Web and linked data conference, dealing with research into Semantic Web and linked data access, integration, inferencing and reasoning. ISWC was held from 23-27th October in Bonn. The institute's Mike Jackson presented GeoTOD-II on behalf of the project as part of the Terra Cognita workshop on linked data and geospatial data. Technical questions on the GeoTOD-II framework focused on the overhead of converting legacy data on-demand per-request and the possibility of applying geo-spatial extensions to its linked data stores and relational databases. A related question arose relating to the UK Cabinet Office recommendations on URIs for location and whether these imply a centralisation that goes against the spirit of linked data. The presentation also prompted discussion on the UK's open data strategy.

The GeoTOD-II framework is available on SourceForge and a demonstration server is available. The framework is currently being used within the JISC-funded ACRID (Advanced Climate Research Infrastructure for Data) project.

For further information, please contact GeoTOD-II.

Software Sustainability Institute Blog | Body scanning and patterned socks

2011-11-18T09:41:17+00:00

By Simon Choppin, a Software Sustainability Agent, (seen here having his feet scanned with the aid of patterned socks).

The Swiss city of Lugano borders Italy, lies on the shores of a crystal clear Lake of the same name and is surrounded by spectacular mountains. The high quality of life, low taxes and agreeable weather has convinced international celebrities and world class athletes to relocate to this Alpine idyll. On the 25-26 of October I briefly joined them, but it wasn’t the weather, lifestyle or even interesting cheeses which attracted me, but the 2nd International Conference on 3D body scanning technologies. The conference packed 60 presentations into an intensive two-day program with a high level of interest from private companies (60% of the attendees were from the commercial sector). The conference focused on body scanning hardware, techniques and applications, of which there were many. Accurate visualisations of facial surgery, large-scale anthropometric surveys, and even custom shoe designs were discussed in depth. Presentations were roughly split into two themes: the hardware which does the scanning, or the software which handles the data.

The hardware takes many forms: handheld scanners, large walk-in booths and of course the now ubiquitous Microsoft Kinect Xbox controller. Three-dimensional information is captured using the time of flight of laser light, tracking the deformation of a pattern on a surface, or by triangulating specific points from different camera viewpoints. As technology improves, the speed of capture and density of data increases, improving the overall quality of body scans, but this brings its own challenges. Captured data exists as hundreds of thousands (or even millions) of discrete data-points, which are essentially three-dimensional co-ordinates. If the scan was taken from multiple viewpoints then each co-ordinate system will be aligned differently, each data point will contain error and the ‘body’ being scanned may have moved or breathed between captures. Software solutions rotate and align these clouds of data to form coherent shapes and surfaces, they smooth noise to minimise error and can even add detail based on high-resolution images of skin texture and blemishes.

The hardware and software approach taken by a company or researcher is very much dependent on their specific objectives. Disney Research are very interested in realistic skin texture and appearance and take a very different approach to 3dMD, a company in the medical sector interested in highly accurate anatomical measurement.

All of the problems and issues discussed at this conference were ones we face in our own research. Learning the plethora of solutions used by the community was both informative and mind-blowing. I had come to this conference as a bit of an outsider, I went to meet people in the community and learn about body-scanning methods and approaches for a new project. As a newcomer I was quickly seduced by the incredible results on display, the fantastic looking images and amazing claims of scan time and performance. It wasn’t until I took a step back and had conversations with delegates that I discovered the very specific nature of many of these techniques. Software is often bespoke and the search for a ‘one size fits all’ solution proved fruitless. Coming back from the conference I’ve been careful to compare our aims with others and look further into their software routines and algorithms. For example, a routine which processes in real-time is of no use if the accuracy of scanning is compromised as a result. A ‘rough’ looking scan with a high level of volumetric accuracy is of much more use to us than one which has been distorted in order to improve appearance.

Unfortunately two days of solid rain meant I didn’t get to enjoy the Alpine vistas and crystal clear waters of Lake Lugano although this introduction to the world of body scanning was enough of an eye opener to compensate. I hope that valuable new contacts will be able to help us in our research and develop our own understanding of this fascinating area.