Infrastructure News Type: Outage Full

Affected Infrastructure: hive.gatech.access-ci.org

Start Date: January 23, 2024, 12:00 p.m.

End Date: January 26, 2024, 5:59 a.m.

PACE Quarterly Maintenance period is scheduled to begin at 6:00AM on Tuesday, 01/23/2024, and is scheduled to conclude by 11:59PM on Friday, 01/26/2024. Please note, as usual, jobs with resource requests that would be running during the Maintenance Period will be held until after the Maintenance Period by the scheduler. During the Maintenance Period, access to all the PACE managed computational and storage resources will be unavailable. Please see the list of activities to be completed, which are posted at https://blog.pace.gatech.edu/?p=7778

Posted: March 20, 2026 • Author: John Coulter

Infrastructure News Type: Outage Full

Affected Infrastructure: xdmod.access-ci.org

Start Date: January 16, 2024, 1:00 p.m.

End Date: January 16, 2024, 11:00 p.m.

ACCESS XDMoD (https://xdmod.access-ci.org/ ) will be unavailable from approximately 7:00AM to 5:00PM EDT on Tuesday January 16th 2024 during a scheduled monthly downtime. The downtime will cause a full outage for both XDMoD and the ACCESS Metrics site. These services should be unavailable for only a couple of minutes despite the full-day downtime. A follow-up message will be sent when the downtime is complete.

Posted: March 20, 2026 • Author: Conner Saeli

Infrastructure News Type: Outage Full

Affected Infrastructure: xdmod.access-ci.org

Start Date: December 19, 2023, 6:00 p.m.

End Date: December 20, 2023, 12:00 a.m.

UPDATE 12/19/23 15:56 EDT: The downtime is complete and ACCESS XDMoD is back up. Thank you for your patience. ACCESS XDMoD (https://xdmod.access-ci.org/ ) will be unavailable from approximately 12:00PM to 6:00PM EDT on Tuesday December 19th 2023 during a scheduled infrastructure update. This will temporarily be a full outage of the service. A follow-up message will be sent when the update is complete.

Posted: March 20, 2026 • Author: Conner Saeli

Infrastructure News Type: Outage Full

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org, expanse-ps.sdsc.access-ci.org

Start Date: December 18, 2023, 3:00 p.m.

End Date: December 19, 2023, 2:30 a.m.

The Slurm scheduler upgrade has been completed on Expanse and the machine is available for use. Slurm was upgraded from version 21.08.8 to 23.02.6. Please note that with the upgrade of Slurm, srun default behaviour has changed. Details of release specific changes are available in: https://github.com/SchedMD/slurm/blob/slurm-23-02-6-1/NEWS (https://urldefense.com/v3/__https://github.com/SchedMD/slurm/blob/slurm-23-02-6-1/NEWS__;!!Mih3wA!CdIUBFjMQW1aL5WriJZf0AW9DInW3G8D99tY-K4oEFYdAWirVTSpm_6et8qGavnPSV87kgRvaigIvXsrQRih$) One change in particular might impact some users as srun is no longer reading SLURM_CPUS_PER_TASK. This meas that the --cpus-per-task value set in the #SBATCH specification will not be automatically picked up by any srun command within the script. Users can either add a specific option to their srun command OR set the following variable before the srun commands: export SRUN_CPUS_PER_TASK=${SLURM_CPUS_PER_TASK} No changes are required if your script was using Intel MPI and mpirun. Please contact us either via the ACCESS ticketing system or via email to consult@sdsc.edu if you have any questions. >>>>>>>> Dear Expanse User, We will have a maintenance period on Expanse 7AM-Midnight (PT), Dec 18, 2023. During this maintenance, we will be updating the Slurm scheduler. We have a reservation in place to prevent jobs from running during this period. The "squeue" output will show "ReqNodeNotAvail, Reserved for maintenance" for jobs that do not fit in the time period before the maintenance begins. These jobs will run after we release the maintenance reservation. Thanks SDSC User Support Staff

Posted: March 20, 2026 • Author: Mahidhar Tatineni

Infrastructure News Type: Outage Partial

Affected Infrastructure: xdmod.access-ci.org

Start Date: December 14, 2023, 2:00 p.m.

End Date: December 14, 2023, 5:00 p.m.

UPDATE 12/14/23 13:39 EDT: ACCESS XDMoD is up now, however some features might not be available for a couple more hours. Thank you for your patience during this time. ACCESS XDMoD (https://xdmod.access-ci.org/ ) will be unavailable from approximately 10:00AM to 1:00PM EDT on Thursday December 14th 2023 during a scheduled infrastructure update. This will temporarily be a full outage of the service. A follow-up message will be sent when the update is complete.

Posted: March 20, 2026 • Author: Conner Saeli

Infrastructure News Type: Outage Full

Affected Infrastructure: anvil.purdue.access-ci.org, anvil-gpu.purdue.access-ci.org

Start Date: October 17, 2023, 1:45 p.m.

End Date: October 17, 2023, 2:55 p.m.

Dear Anvil user, Anvil nodes experienced a brief outage this morning. The problem is resolved and nodes are online now. Please check the status of your job and resubmit if necessary.

Posted: March 20, 2026 • Author: Ruyi Li

Infrastructure News Type: Outage Full

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org, expanse-ps.sdsc.access-ci.org

Start Date: September 29, 2023, 7:05 p.m.

End Date: October 1, 2023, 1:30 a.m.

The Expanse home directory server issues have been resolved and the machines is back in production and available for use. >>> We are seeing the NFS server issues again on Expanse and this is causing login issues. We are working with the vendor on identifying the source of the problem and will update once we have resolution. In the interim we are putting in a system reservation to prevent new jobs from starting on Expanse.

Posted: March 20, 2026 • Author: Mahidhar Tatineni

Infrastructure News Type: Outage Full

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org

Start Date: September 28, 2023, 8:00 a.m.

End Date: September 28, 2023, 9:30 p.m.

Update: The home directory issues on Expanse have been resolved and the system is available for logins and use now. >>> The SDSC Expanse home directory servers had issues overnight and that is leading to login problems. We are looking into the issues and will update once they are resolved.

Posted: March 20, 2026 • Author: Mahidhar Tatineni

Infrastructure News Type: Outage Full

Affected Infrastructure: tickets.access-ci.org

Start Date: September 27, 2023, 6:25 p.m.

End Date: September 27, 2023, 7:02 p.m.

Jira Service Management ticketing system is non-responsive again. We are working with the support to resolve this and avoid this in the future. We are not able to give an ETA at this time.

Posted: March 20, 2026 • Author: Dinuka De Silva

Infrastructure News Type: Outage Full

Affected Infrastructure: tickets.access-ci.org

Start Date: September 27, 2023, 1:00 p.m.

End Date: September 27, 2023, 3:19 p.m.

Dear ACCESS colleagues, the Jira Service Management ticketing system is non-responsive in certain geographical regions. We have submitted a support ticket but have no ETA on the resolution

Posted: March 20, 2026 • Author: Dinuka De Silva

Infrastructure News Type: Outage Partial

Affected Infrastructure: identity.access-ci.org

Start Date: August 21, 2023, 1:34 p.m.

End Date: August 21, 2023, 6:00 p.m.

Update as of 08-21-2023 1:00 pm Central The DUO multi-factor services appears to be working normally again. Original News The ACCESS Multi-factor Authentication (MFA) service provided by DUO is experiencing slowness or failure, affecting all logins using ACCESS username and password. The vendor is aware of the problem and working to resolve it. The vendor is posting outage updates here (https://status.duo.com/incidents/rw7g0q7ztj8f) and here (https://status.duo.com).

Posted: March 20, 2026 • Author: John-Paul Navarro

Infrastructure News Type: Outage Full

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org, expanse-ps.sdsc.access-ci.org

Start Date: August 14, 2023, 2:00 p.m.

End Date: August 15, 2023, 6:59 a.m.

We will have a maintenance period on Expanse 7AM-Midnight (PT), Monday, August 14, 2023. There is a reservation in place to prevent jobs from running during this period. The "squeue" output will show "ReqNodeNotAvail, Reserved for maintenance" for jobs that do not fit in the time period before the maintenance begins. These jobs will run after we release the reservation. The SDSC Expanse portal and the SDSC Expanse Globus collections will also be unavailable during this maintenance period.

Posted: March 20, 2026 • Author: Mahidhar Tatineni

Infrastructure News Type: Outage Full

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org

Start Date: May 8, 2023, 3:00 p.m.

End Date: May 10, 2023, 6:59 a.m.

We will have maintenance scheduled on Expanse starting 8AM (PT) May 8 through May 9, 2023. During this maintenance several upgrades will be performed. Details of the upgrades and the process are below: - The operating system will be upgraded from Rocky Linux 8.5 to Rocky Linux 8.7. The OFED version will be updated to 5.8.1.1. In addition, the firmware on all nodes will be updated. - The drivers on the GPU nodes will be updated to version 515.65.01 (CUDA 11.7). - The Lustre client version will be updated to version 2.15.2. - A new software stack build using Spack 0.17.3 will become available and will be the default (available as modules cpu/0.17.3b or gpu/0.17.3b). - The original software stack will continue to be available and will work with the new OS and OFED versions. The old software stack can be accessed by using "module load cpu/0.15.4" or "module load gpu/0.15.4". - We will be making the new software environment available in advance of the maintenance and will send out an update once it is available. - Jobs submitted before the upgrade without specifying the version of cpu (0.15.4) or gpu (0.15.4) must be cancelled and resubmitted after the maintenance to ensure that the right environment is picked up for their jobs (since the defaults will change to 0.17.3b). - We are reserving nodes for multiple days given the large number of updates involved. Jobs that will not fit in the time window before the maintenance will be pending and held before the upgrade. - After our service nodes are upgraded and the Lustre update is completed, we will look to put fully upgraded nodes back into the queue as possible to help mitigate the long downtime. Please submit a ticket via ACCESS Support (https://support.access-ci.org (https://urldefense.com/v3/__https://support.access-ci.org/__;!!Mih3wA!HahR9U4iAF_EKq7SJwqOxyu08ng7j6AyntEFwNvdhS50YdBqK4fndYVUklk9YLomcjxQK6kJjjYviBVeeeVl$)) or SDSC's local ticketing system (consult@sdsc.edu (mailto:consult@sdsc.edu)) if you have any questions. Thanks, SDSC User Services Team

Posted: March 20, 2026 • Author: Mahidhar Tatineni

Infrastructure News Type: Reconfiguration

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org

Start Date: April 26, 2023, 6:00 a.m.

End Date: April 27, 2023, 11:30 p.m.

Update 04/27/2023: We have restored the overlay functionality on all Expanse nodes. Original message: In response to a security advisory, we have disabled the overlay kernel module on all Expanse nodes. This will prevent the use of some Singularity containers and possibly CVMFS mounts on Expanse. We are working on a mitigation approach and will update once this functionality is restored. Please send a ticket via ACCESS support (https://support.access-ci.org (https://urldefense.com/v3/__https://support.access-ci.org__;!!Mih3wA!DwrHmYkjt_poeQCAi67dZKlq73yxtpBB5_DWxOIFYI2TQKhrssmOLNDvj573Rwg2DVkkiWov67YTxMaz1S2M$)) or SDSC's support email (consult@sdsc.edu) if you have any questions. Thanks, SDSC User Services Staff

Posted: March 20, 2026 • Author: Mahidhar Tatineni

Infrastructure News Type: Outage Full

Affected Infrastructure: ranch.tacc.access-ci.org

Start Date: March 26, 2023, 1:00 p.m.

End Date: March 26, 2023, 7:00 p.m.

TACC network infrastructure will not be available from 8:00 AM to 2:00 PM (CDT) on Sunday, 26 March 2023. Network maintenance will be performed during this time. Please submit any questions you may have via the TACC User Portal. https://portal.tacc.utexas.edu/tacc-consulting

Posted: March 20, 2026 • Author: Mark Brueschke

Infrastructure News Type: Outage Full

Affected Infrastructure: lyric.uky.access-ci.org

Start Date: February 24, 2023, 9:00 a.m.

End Date: February 25, 2023, 12:00 a.m.

The KyRIC system will be offline on Friday, 2/24/2023, 3AM-6PM ET due to maintenance of the storage system. All nodes and data operations will not be available in that period. We aim to bring the system back on the same day. KyRIC Staff

Posted: March 20, 2026 • Author: Satrio Husodo

Infrastructure News Type: Outage Full

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org

Start Date: February 21, 2023, 4:00 p.m.

End Date: February 22, 2023, 1:00 a.m.

We will have a maintenance period on Expanse 8AM-5PM (PT), Feb 21, 2023. During this maintenance, we will work on the NFS home filesystems. We have a reservation in place to prevent jobs from running during this period. The "squeue" output will show "ReqNodeNotAvail, Reserved for maintenance" for jobs that do not fit in the time period before the maintenance begins. These jobs will run after we release the maintenance reservation. Thanks SDSC User Support Staff

Posted: March 20, 2026 • Author: Mahidhar Tatineni

Infrastructure News Type: Outage Full

Affected Infrastructure: faster.tamu.access-ci.org

Start Date: February 8, 2023, 3:00 p.m.

End Date: February 9, 2023, 2:00 a.m.

FASTER will be unavailable from 9a to 8pm for software and hardware maintenance.

Posted: March 20, 2026 • Author: Francis Dang

Infrastructure News Type: Outage Full

Affected Infrastructure: stampede2.tacc.access-ci.org

Start Date: January 28, 2023, 7:19 p.m.

End Date: January 29, 2023, 5:59 a.m.

Stampede2 is currently experiencing issues with the /scratch filesystem and queues will remain down for a prolonged duration as the issues are investigated.

Posted: March 20, 2026 • Author: Matthew Edeker

Infrastructure News Type: Outage Partial

Affected Infrastructure: xdmod.access-ci.org

Start Date: November 30, 2022, 7:01 p.m.

End Date: December 13, 2022, 4:59 a.m.

The work to bring the new Allocations information online has now been completed. ---------------------------------------------------------------------------------------------------------- ACCESS XDMoD (https://xdmod.access-ci.org/ ) may experience some temporary performance degradation while work is done to bring the new Allocations information online. Chart load times may be slower than usual. Another notice will be sent when the work is complete.

Posted: March 20, 2026 • Author: Ryan Rathsam