Live Status Updates

CURRENT ISSUES


3rd August 2015

PCF staff are currently decommissioning MERRI to clear the way for the delivery of the new system.  As mentioned below we will be unplugging Infiniband cables and so Avoca is at risk whilst this process is completed.   We will be taking the utmost care and do not expect any impacts, but we flag this just in case.

 

UPCOMING ISSUES


DECOMMISSIONING MERRI CLUSTER - GPFS AT RISK

3 August 2015

In the process of decommissioning the old MERRI cluster, the Facility team will need to unplug a number of Infiniband cables.  Although we will excercise the highest degree of caution, we will have AVOCA  in  an  'at risk' modes just in case on of its cables is impacted.  NOTE that BARCOO should be reasonably insulated from this, as its cables are on a separate switch.

DECOMMISSIONING MERRI CLUSTER - NEW DATE

New Date: 2 August 2015

We are pleased to be able to inform users that the procurement process for our new system has been finalised.

To this end, MERRI will be decommissioned at 23:45 (11:45 pm) on Sunday 2 August, 2015.  The majority of users whom this impacts have already been informed, but BARCOO users will find that for a short time,  the queue on BARCOO will be more full.

We will receive delivery of the new system very soon after the decommissioning of MERRI and we will keep users posted on the status of commissioning the new cluster. 

DECOMMISSIONING MERRI CLUSTER - POSTPONED

July 2015

Further delays in the procurement process have meant that the best course of action for the proposed decommisioning of the MERRI cluster is to delay this until the procurement has been properly settled. We apologise for this action, but we should be able to give users a good four weeks of notice before MERRI is no longer available.

On a positive note, this will decrease the time between the old system being decommissioned and the new one being in production.

DECOMMISSIONING MERRI CLUSTER - BARCOO USERS PLEASE NOTE

BARCOO users should not that there will be an extra demand on the BARCOO cluster due to the decommissioning of MERRI and the delay in the arrival of the new system.  We will limit large numbers of jobs for any individual or group so that a more fair share approach is maintained during this interim period.  As stated below, we anticipate the new system to be in production somewhere in mid-August.

DECOMMISSIONING MERRI CLUSTER

More Information:  18 June 2015

The proposed decommisioning of the MERRI cluster will proceed as planned.  However, due to delays in the procurement process it is anticipated that the new system will not be online until mid-August.  MERRI users will be accorded time on the BARCOO cluster in the interim.  Specific details for access will be provided to users directly.

DECOMMISSIONING MERRI CLUSTER

30 June 2015 : 11:00 pm

VLSCI is currently (May 2015) going through a tender process for a new x86 cluster.  The new system will replace the current MERRI cluster, and should replace the capacity of both the old BRUCE and the current MERRI clutsters.  For this to happen, the MERRI cluster will be shutdown and removed from VLSCI's server room.

To this end, MERRI will be officially decommissioned on the 30 June 2015 at 11:00 pm.  Thereafter, MERRI users will need to use the BARCOO system until the new cluster becomes operational in early Q3.

More information regarding this change will come shortly.


PREVIOUS ISSUES


Unplanned power outage

9 July 2015: 6:20pm

The batteries have been replaced and the systems have now been returned to service.

However, we still need to monitor the performance of the storage system as further work may be necessary, so we have placed a provisional reservation on MERRI, BARCOO and AVOCA for 1pm on Monday 13th July to permit work if necessary.  If this work is not needed then we can release that reservation and longer jobs will be able to start.

Apologies for this inconvenience!

9 July 2015 : 10:30am

Outage update: Power has been restored and we are awaiting the replacement of a battery for the failing component of the storage system.  Once this system has been tested, we hope we can start returning systems to service.

We will let you know of our progress towards the end of the day.

8 July 2015 : 3:00pm

A key component for the storage system has failed to return to service, and we are waiting on the vendor for a solution.  All user data is unaffected.

Unfortunately this makes it extremely unlikely that our systems will return to service today.

Once again we sincerely apologise for the disruption this has caused.  We are investigating the circumstances of this outage and ensuring it does not happen again. 

8 July 2015 : 11:30am 

Unfortunately ongoing construction work at the data centre hosting VLSCI equipment has resulted in a complete power outage.

Our team have been working on site since early this morning to ensure services are restored as quickly as possible. These are complex systems that do not like suddenly having the power turned off.

We will let you know of our progress towards the end of the day.

The help system (help@vlsci.unimelb.edu.au) in now available and queued e-mails are slowly trickling in.

Currently the help system is also unavailable.  Any sent e-mails will be queued and will be processed by the help system once it is available. If you need to contact us more urgently, call or email Andrew Isaac on 90355686, aisaac@unimelb.edu.au.

We apologise for the nuisance, please be assured that all lost quota will be refunded. 


ITS ELECTRICAL WORKS - FACILITY SHUTDOWN - CANCELLED

6-7 JUNE 2015

ITS  will be doing major electrical works  in the building that hosts VLSCI's systems and will require the facility to be completely powered down.  There will be no services available during this time, and jobs can only be queued once the systems have been restored.

Our apologies for this inconvenience, and thank you for you patience.


AVOCA MAINTENANCE Completed

5 June 2015 : 16:20

VLSCI is please to announce that the AVOCA upgrade has successfully completed and we are now running at RH6.6, and V1R2M3 for the BlueGene release.  Most of the general user software has been rebuilt, and we kindly request users to send a ticket to the help desk if they have any issues.

It is HIGHLY recommended that users who run their own code recompile their code for the upgraded system.

AVOCA MAINTENANCE Update

4 June 2015

We have, after a few minor teething issues, succesfully passed the first stage of our AVOCA upgrade.  There are still some steps to go before the system is available again.  We will keep you posted as more successful stages are passed.

Thanks again for your patience during this maintenance preiod.

AVOCA MAINTENANCE

1 June 2015 : 10:00am  to 8 June 2015 : 12.00

AVOCA needs to have updates to its operating systems to continue to be supported by IBM. Unfortunately this necessitates taking down all of AVOCA. We anticipate a week long shutdown, but hope that the maintenance goes smoothly, for a shorter maintenance break. Users will be updated as we move through successful stages.

Our apologies for this inconvenience, and thank you for you patience.


VLSCI USER PORTAL (KARAAGE) UPGRADE UPDATE

The account manager interface has a new look, and that the URL has changed to

https://my.vlsci.org.au/karaage/

We believe we have updated all links to the new URL, but please inform our help desk if you encounter
any problem while accessing it.

UPDATE : 06 May 2015 15:00

VLSCI is pleased to announce that the storage updates have completed successfully and we are now ready to resume services. 

UPDATE : 06 May 2015

The firmware updates on our storage systems have not been successful and we are waiting for the vendor to fly in replacement parts.  We WILL endeavour to get back to normal services as soon as possible, but currently there is no time frame that we can confidently predict.

Again, our sincerest apologies for this inconvenience, and thank you for your patience.

UPDATE : 05 May 2015

Unfortunately our account management update has not gone as smoothly as we had hoped.  As this is a critical part of our services, we will be extending this shutdown to 10 am Wednesday 6th May. We will also use this opportunity to do some further minor repairs and updates to our systems.

Our apologies for this inconvenience, and thank you for you patience.

ITS NETWORK MAINTENANCE - FACILITY SHUTDOWN

01 May 2015 : 1 pm  to  05 May 2015 : 12:00 pm

PLEASE NOTE THAT THERE WILL BE NO HELP DESK AVAILABLE.  WE WILL RESPOND TO HELP TICKETS  ONCE THE SYSTEMS ARE RESTORED.

ITS will be performing network upgrades at the building hosting VLSCI's systems.  There will be no access to systems in general but unfortunately this impacts VLSCI's internal networks so we will need to shutdown all computational nodes for all systems.  Other services which will be impacted are our help system, user portal (Karaage) and possibly the VLSCI website.

Jobs queued in the system will be started once the network systems are restored.

Our apologies for this inconvenience, and thank you for you patience.

ITS NETWORK MAINTENANCE - FACILITY SHUTDOWN

UPDATE: 30 Apr 2015

As the ITS electrical works have been cancelled the following shutdown has been extended to the 5 May to enable some systems to be updated.  The update of the user account management system (Karaage) will happen during this shutdown.


VLSCI USER PORTAL (KARAAGE) UPGRADE

4 May 2015

VLSCI will be updating the user portal system.  The user portal will be in a read only mode until the upgrade is completed.  This upgrade will be performed during the above outage.

More information regarding this upgrade will be available soon.


 

VLSCI FILE SYSTEMS AT RISK MODE

19 March 2015

The failed SAN switch was replaced yesterday and systems are no longer running at risk.

13 March 2015

One of the SAN switches has failed and so we are running on the failover switch. The new hardware is due to arrive early next week, but in the interim, there is a small risk that the remaining SAN switch will also fail, which would result in no access to the storage systems.

Our apologies for this inconvenience, and thank you for you patience.

AVOCA UNSCHEDULED MAINTENANCE

18 March 2015 - 5pm

It was necessary to completely restart the BlueGene/Q control stack in order to restart the underlying DB/2 database.  Unfortunately this meant the loss of all running jobs. We have refunded quota and returned Avoca to service.

Our apologies for this inconvenience, and thank you for you patience.

18 March 2015 - 11am

There is a current problem with the SLURM scheduler on AVOCA. Unfortunately this means that no jobs can be queued at present. The PCF team is investigating the issue and will repair it as soon as possible.

In the case that the running jobs are lost during the process, quota will be refunded, as is usual.

Our apologies for this inconvenience, and thank you for you patience.

 


 

AVOCA  MAINTENANCE

23 February 2015 11:30

Avoca is back in service after its scheduled maintenance.

23 February 2015 8:00 to 12:00

The database process on AVOCA needs to be restarted and this means we need to stop the control processes for the BlueGene/Q racks.  This necessitates a small outage window for running jobs on AVOCA.   During this time AVOCA users will be able to read and transfer files but will not be able to submit new jobs to the queue.

We will inform users as soon as the system is available again.

The VLSCI  team thanks you for your patience during this maintenance period.

AVOCA UNSCHEDULED MAINTENANCE

23 December 2014 18:35

Avoca is back in service.

23 December 2014 17:45

The Avoca login node crashed and spontaneously rebooted this evening and appears to have a hardware issue. Sadly this means all running jobs will have been lost.

We are investigating and will try and return Avoca to service as soon as possible.

AVOCA MAINTENANCE COMPLETED

02 December 2014 11:20

AVOCA is now ready for use.  Our apologies that the repairs to the AVOCA data base took longer than expected.  The actual problem was a kernel bug in the release of the RedHat kernel we were using.  We've rolled back to an earlier kernel which was known to work.

Tthank you for you patience during this unexpected extension to the outage.


AVOCA MAINTENANCE

Update: 14:00

We have encountered a bigger issue with our data base system on AVOCA than initially thought. PCF is working on resolving the problem to a level where we can bring users back on the system as soon as possible.

Thank you for you patience in the interim.

01 December 2014 : 09:00 - 01 December 2014 12:00

PCF will do a short maintenance on AVOCA's data base systems on Monday 01 December.

We expect the outage to be relatively short and users will be contacted promptly when the system is back online.

Thank you for your patience during this brief outage.


SCHEDULED MAINTENANCE COMPLETED 10 Nov 2014

AVOVA maintenance is now completed.  We thank users again for their patience during this period.

PCF requests that users recompile their own code with the current libraries if they have any issues running their programs.

There are still some minor issues with some software running on AVOCA, and although we are aware of this, please send a ticket to our helpdesk when you encounter issues.  This will help us track down common problems.


FACILITY MAINTENANCE - FULL SHUTDOWN

05 November 2014 : 09:00 - 05 November 2014 15:00

An urgent repair to the UPS system, through which  the PCF server room is powered needs to be made on this day.  Unfortunately this will not just affect clusters but also all our services - helpdesk, web, RAS applications, etc.  Every effort will be made to restore these services ASAP, but we will have a broad window to cover contigency plans.

In light of this full shutdown, the MERRI and BARCOO scheduled maintenance will be pushed back to the 4th of November as detailed below.

The PCF team thanks you for your patience during these maintenance periods.


MERRI SCHEDULED MAINTENANCE

04 November 2014 : 07:00 - 06 November 2014 23:59

There will be a full outage for MERRI to upgrade system software.  The outage is expected to last all day, and users will be informed if the maintenance is completed earlier.

BARCOO SCHEDULED MAINTENANCE

04 November 2014 : 07:00 - 06 November 2014 23:59

There will be a full outage for BARCOO to upgrade system software.  The outage is expected to last all day, and users will be informed if the maintenance is completed earlier.

AVOCA SCHEDULED MAINTENTANCE

05 November 2014 : 07:00 - 07 November 2014 23:59

There will be a full outage for AVOCA to upgrade system software.  The outage is expected to last all day, and users will be informed if the maintenance is completed earlier.

The PCF team thanks you for your patience during these maintenance periods.


 


29th September 2014 - 16:00

The Barcoo login node is having some stability issues that was preventing access, we have returned it to service.

Running jobs were not affected and we will continue to monitor the situation.

Update: We have tracked this down to a known kernel issue that was already solved, but needed a full power cycle to settle properly after the update.

AVOCA UNSCHEDULED MAINTENANCE

Update: 15:00

AVOCA is back online and ready to accept jobs.

Thanks again for you patience.

03 September 2014 : 11:45

There has been an unforeseen software glitch on AVOCA and we are currently repairing the issue. Unfortunately that involves stopping the scheduler, so new no jobs will run.  Any currently running jobs which are affected by this glitch will have their quota returned.

Our apologies for this inconvenience, and thank you for you patience.


MERRI login node issue

Update: 15:00

The reboot was successful and all services

19 August 2014: 14:50

An issue with stale file handles on the login node meant that the node needed to be rebooted.

The reboot should only be a matter of minutes.

No user jobs or files have been lost during this intervention.

Apologies for this inconvenience and we thank you for your understanding.


AVOCA Maintenance of 2 mid-planes

Update: 17:00

Maintenance  is completed, and the system is fully operational.

Thank you for you patience in the interim.

11 August 2014: 10:00

There will be some urgent maintenance made on Monday on two mid-planes of AVOCA.  This will put AVOCA in a reduced capacity.  The maintenance window will start at 10:00 and we expect the maintenance to last one day.

Users can log in as normal to the system, running and queued jobs will not be affected.

Apologies for this inconvenience and we thank you for your patience in the interim.


BRUCE Decommissioned

1 July 2014

The BRUCE system has now been stopped for new jobs.  Users have one month to remove their data from the disk system, after which time, the system will be donated to NECTAR.

Bon Voyage BRUCE!


MERRI fully operational

27 May 2014

VLSCI is pleased to announce that all necessary inifinband cables for  MERRI have been delivered and installed.

MERRI is now in full production.

Thank you for your patience and understanding for this extended situation.


AVOCA fully operational, MERRI at 93% capacity

14 April

VLSCI is pleased to announce that AVOCA  now has all required cabling in place and is no longer running at risk. 

MERRI is currently still running with 77 nodes out of the full rack.  We hope to have the remaining nodes connected when new inifiniband cables are delivered.  Currently there is no firm ETA, but we expect cables to arrive within the next few weeks.

We thank you for your patience in the interim.


GPFS Hardware Problems Resolved

14 May 2014

Update:  8:30

All disk rebuilds have completed and things have returned to normal. The schedulers have been resumed and new jobs will now run.

Thank you for your patience during this brief outage.


GPFS Hardware Problems

13 May 2014

Update: 17:33

The current situation is that affected disks are currently rebuilding. As this will take around 12 hours and GPFS performance will be reduced during the rebuild, we will not restart the scheduler. However, logins will be permitted so that users can prepare jobs for when the system is fully restored.

We will keep people posted as we get back to full production.

Thank you for your patience during this unscheduled break.


GPFS Hardware Problems

13 May 2014

Our disk system has suddenly lost a large number of disks.  We have halted  the scheduler on AVOCA, MERRI and BARCOO and to ensure that any possible file corruption is at a minimum, logins have also been blocked.

We will keep people posted, and hope that the systems will be available again shortly.

Thank you for your patience during this unscheduled break.


AVOCA Maintenance finished

28 April 2014

Maintenance end  at 18:10

AVOCA is now back on line and ready for use.  After some firmware updates and some minor network changes we hope that the issues affecting AVOCA have now been resolved. 

Queued jobs are now running, and any lost jobs will have their quota refunded.

Thank you for your patience during this unscheduled break.


AVOCA Maintenance

28 April 2014

Maintenance start at 12:30

We will take AVOCA down for maintenance effective immediately to try to solve the issues that are currently detaching the I/O nodes from the system.  We hope to have the system up again later this afternoon. 

Please note that no queued jobs will be lost, however,  current user jobs will be. Lost jobs will of course, be refunded their lost quota.

Apologies for this inconvenience and we thank you for your patience in the interim.


SOME AVOCA NODES unavailable

27 April 2014

AVOCA is still experiencing some issues with some nodes.  Unfortunately this will mean that AVOCA is not 100% available.  There will be some maintenance done on these nodes at 11:00 am Monday 28 April.

Please stay tuned to this live status update for more information.

Apologies for this inconvenience and we thank you for your patience in the interim.


SLURM issues on  AVOCA - solved

24 April, 12:00 pm

The issue has now been resolved.  Lost jobs will have their quota refunded.

Apologies for the inconvenience, and thank you for your patience.


SLURM issues on  AVOCA

24 April, 11:00 am

The SLURM scheduler demon died earlier this morning.  We are currently working on this issue and will get the system back to normal as sson as possible.

We thank you for your patience in the interim.


Short Maintenance Break for AVOCA completed

23 April, 12:00 am

The maintenance break on AVOCA is now over.  Thank you for your patience.


Short Maintenance Break for AVOCA

23 April, 10:00 am

There will be a short maintenance break for AVOCA this morning to alleviate some issues which have escalated over the Easter break.

Apologies for the short notice.


AVOCA AT RISK

06 March

AVOCA is currently still running in an "at risk" mode while we wait for replacement infiniband cables. Altough we don't anticipate problems, we advise people to note that as we do no have redundancy at present we could loose an interface to AVOCA if any of the current cables fail.

MERRI is currently  running with 76 nodes out of the full rack.  We hope to have the remaining nodes connected when new inifiniband cables are delivered.

We thank you for your patience in the interim.


UNIVERSITY OF MELBOURNE NETWORK ISSUES

27 February, 12:30pm

We have been pursuing network issues that break SSH connections into our systems from the University of Melbourne network only.  It appears this was caused by a UoM firewall upgrade on Friday 21st which introduced a bug in the Cisco code.   UoM ITS will be downgrading to the previous stable version tonight, Thursday 27 at 7pm.


MAINTENANCE

UPDATE: 28 February, 16:00

We have recovered another 37 nodes of MERRI, but still have issues with some of the IB cables.   Some of these will need to be replaced.  AVOCA is still running at risk. 

Again, we apologise for inconvenience in the reduction of our services.

UPDATE: 26 February, 18:45

AVOCA is now back and is available for logging in and queueing jobs.  However, it is currently running in a non-redundant mode so please be aware it is running at risk until its new cables arrive tomorrow morning.

UPDATE: 26 February, 16:00

MERRI is now has 30 nodes up and is available for logging in and queueing jobs.  As we fix the issues with the cables, more nodes will come online.

Again, we apologise for this inconvenience.

UPDATE: 26 February, 8:00

The GPFS updates have been successfully completed.  BARCOO is ready for production again.

There are still some minor issues with the cabling of the new infiniband switch. We are working on this and will keep users informed as to progress

Critical Updates for GPFS system

Maintenance time: 25 February 2014

There have been some critical updates released for the GPFS system.  This will affect the AVOCA-MERRI-BARCOO systems. We will apply these and use this opportunity to replace the inifiniband switch for MERRI and AVOCA.

Systems affected: AVOCA, BARCOO and MERRI.

Start time: 8:00am

Thank you for your patience during this maintenance break.


10 February 2014: System login problems - resolved

1:05 pm Update:

The cable has been replaced, and users can log back into MERRI's frontend.  We will continue to monitor the status of MERRI to ensure stability

10 February 2014: System login problems

12:45 Update: We suspect a broken cable and are currently fixing the login problem to MERRI.

Please note that user jobs are not affected.

11:30: MERRI is currently experiencing some problems.  We are fixing this problem now. Apologies for the inconvenience.

BARCOO and AVOCA are now available.

We are pleased to announce that the storage maintenance  has been completed successfully, as has the general maintenance on AVOCA, MERRI and BARCOO.

Please do email help@vlsci.org.au if you are experiencing any problems with the systems.

Thank you for your patience during this maintenance break.


Maintenance time: 28, 29 and 30 January 2014

There will be some general maintenance done on the AVOCA-MERRI-BARCOO infrastructure on January 28, 29 and 30th.  During this time none of  AVOCA, MERRI or BARCOO will be available.  Any submitted jobs will be queued until the maintenance is completed.

The maintenance window will start at 8:00 am.  It includes:

    Update of HSM storage software
    Firmware updates on the HSM  Tape Library
    Firmware updates on the fibre channel infrastructure
    General maintenance on all clusters
    Significant work on the infiniband infrastructure

Any changes to the dates, times and affected systems will be announced as soon as possible.

Please note that BRUCE is not affected by this maintenance break.


17 January 2014 - AVOCA status update

5:50 pm

The cool change has now arrived and systems are stable.  We are now bringing AVOCA fully back on line.

Thank you all for your patience while we struggled with this heat event.  As mentioned below,  a permanent fix is being put into place this weekend so that we will be less affected by extreme temperatures in the future.

3:00 pm

After further monitoring we have been able to bring up a second midplane so we are now running Avoca at 16,384 cores.  We will continue to monitor as the cool change arrives.

11:30 am

We believe the cooling system is now stable enough for us to allow longer running jobs to run on the single midplane of AVOCA and have now instructed the scheduler to do that.  We will monitor the situation as the day progresses.

Orders for the remediation of the cooling systems has been placed, the contractor has collected the necessary parts and we expect works to begin on Saturday.


16 January 2014 - AVOCA status update

3:15 pm

We have made the temperatures for water cooling on AVOCA fairly stable at this point, but the situation with the chillers is not  yet properly resolved.  We will cautiously open up one midplane of AVOCA (8192 cores!) and let short  jobs resume.  We will monitor the situation, but we think we can keep this 1/8 of AVOCA running through to the end of the hot spell.


15 January 2014 - MERRI and BARCOO back in production

10:00 am

MERRI and BARCOO have been returned to service.   We are still currently investigating when Avoca can be returned to service


14 January 2014 - AVOCA shutdown due to extreme heat

6:15 pm

AVOCA will remain shut down for tonight, MERRI and BARCOO will stay paused, jobs may be submitted but no new jobs will start.    BRUCE is unaffected by this as it is in a separate data centre.

3:25 pm

Technicians are currently looking into solving the problems with chillers.  In the meantime, AVOCA has been completely shutdown.  MERRI and BARCOO will not accept new jobs at present.

14 January 2014 - MERRI and BARCOO at risk

1:35pm

Due to the problem with the chillers which have put AVOCA offline, we also have MERRI and BARCOO at risk to be suddenly shut down.  At present they are continuing to operate with current jobs but the schedulers are paused for the moment.  We will try to give enough notice about the possible shutdown of these systems. 

14 January 2014 - AVOCA shutdown due to extreme heat

1:25 pm

The current temperatures in Melbourne have been too much for the chiller systems which sit on the roof of the ITS building where AVOCA is housed.  The water inlet temperatures have exceeded the safe limit for keeping AVOCA operational, so we have stopped all jobs at the moment.

All lost jobs will be refunded quota and every effort will be made to try to get systems back to a usable state.  However as the heatwave is forcast for a few days, we will err on the side of caution to make sure AVOCA is usable after this heatwave has passed.


11 October 2013 - Merri Issues - resolved

The hardware fault on MERRI has now been repaired, and Merri is back on line.

Jobs are running, and any lost jobs will be refunded.


11 October 2013 - Merri Issues

Maintenance/Outage times:  At risk from 9:00; Maintenance from 10:00

We currently have an issue with failing hardware in the management node for the MERRI cluster.   We have identified the probelm and are effecting repairs currently.  No new jobs can be submitted at this time.  We will aim to try to keep as many of the current jobs running through the repairs as possible, however there is a risk that they will be lost.

Unfortunately this issue has  meant that the system  lost jobs yesterday. As is our policy the SU for these  and any lost jobs will be refunded.


10 October 2013 - GPFS Issues - Status Update

Initial tests of the latest release of the GPFS client have shown to fix the issue mentioned on 10 October.

Software updates are being rolled out non-disruptively on all systems.  The next status update on this issue will be given when updates are finalised.


1 October 2013 - GPFS Issues - a work around

Maintenance/Outage times: Until further Notice

There is currently a problem with  GPFS  that sends an error to running jobs to say that the job can't write to a file.  This effect has been noticed sporadically across a number of projects using different programmes.

Follow up solution:

The solution to this is to run the job from your project's  /scratch area.  SInce this area is project based, it is important that if your project does not currently have a scratch area that the project manager requests that it is created. 

To request scratch space, (project managers) please go to https://my.vlsci.org.au and log into the system.

Go to the my projects link and select  your project from the list.  Under the  Management Tasks heading there is a  Request Scratch Space link.  Click on this to start the process.

(NOTE: As jobs running from the scratch area generally run faster, moving to /scratch is more than just  a good practice in general.)

Information about /scratch is also found at more about storage on merri, barcoo and avoca.