Infrastructure News Type: Outage Full

Affected Infrastructure: tickets.access-ci.org

Start Date: March 25, 2025, 6:29 a.m.

End Date: March 25, 2025, 6:49 a.m.

Users in some regions are currently experiencing a 502 Bad Gateway error when trying to access the JSM ticketing system Customer Portal. This issue is impacting both customers and agents in those regions. We are waiting for an update from Atlassian support and will provide further information as soon as we hear from them. https://jira-service-management.status.atlassian.com/incidents/vy4wm8b5v0yh

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: tickets.access-ci.org

Start Date: March 24, 2025, 12:39 a.m.

End Date: March 24, 2025, 4:56 a.m.

Hi Everyone, We are encountering some errors in the automation executions and as a result, the notifications for new tickets or ticket edits may not go out. Other than that, there's no impact on managing tickets or queues. Please bear with us while we are troubleshooting this with Atlassian support. We will keep you posted as we know. Thank you

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: xdmod.access-ci.org

Start Date: March 21, 2025, 10:28 a.m.

End Date: March 21, 2025, 2:55 p.m.

We are investigating an issue with connectivity to xdmod.access-ci.org and metrics.access-ci.org. Resolved 9:56 AM EDT

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: registry.access-ci.org

Start Date: March 19, 2025, 1:00 p.m.

End Date: March 19, 2025, 1:30 p.m.

On March 19, 2025, the ACCESS User Registry (https://registry.access-ci.org/) will experience a total outage for approximately 10-15 minutes. During this time, the database used by the ACCESS User Registry will be reconfigured to use "utf8mb4" as the character encoding for several user attributes such as name and organization. This update is necessary to correctly display non-Latin characters. During the outage, visitors to https://registry.access-ci.org/ will be redirected to this infrastructure news notice. Users will not be able to register for ACCESS accounts, make changes to their existing ACCESS accounts, or create/modify OIDC client registrations. Logging on to other ACCESS websites should not be affected. For questions or concerns with this update, please contact help@cilogon.org (mailto:help@cilogon.org) or open an ACCESS Help Ticket (https://access-ci.atlassian.net/servicedesk/customer/portal/2/create/30).

Posted: March 20, 2026

Infrastructure News Type: Degraded

Affected Infrastructure: tickets.access-ci.org

Start Date: March 19, 2025, 12:45 p.m.

End Date: March 20, 2025, 1:00 p.m.

Some users experiencing a 'site maintenance' message when attempting to access Confluence and Jira on some APAC based sites. (https://jira-service-management.status.atlassian.com/incidents/bt1p90s32742) We have partially restored functionality to some of the affected Jira Work Management, Jira Service Management, and Jira Cloud customers. We will continue to work on the fix and share more information as it becomes available.

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: kerberos.access-ci.org

Start Date: March 18, 2025, 6:00 p.m.

End Date: March 18, 2025, 7:00 p.m.

There will be DNS work to transition to the new Kerberos replica server hosted at PSC. No service outage expected as hosts should failover to the other replicas.

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: identity.access-ci.org

Start Date: March 18, 2025, 1:30 p.m.

End Date: March 18, 2025, 1:30 p.m.

On March 18, 2025, the ACCESS Identity Provider (https://idp.access-ci.org/idp) was updated to address a Tomcat vulnerability that was being actively exploited in the wild.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org, expanse-ps.sdsc.access-ci.org

Start Date: March 17, 2025, 4:00 p.m.

End Date: March 25, 2025, 12:00 a.m.

Dear Expanse User, The Expanse Lustre filesystem issues have been resolved and the filesystem is back in production use. Thank you for your patience through the long outage. We will continue to monitor the filesystem and follow up as needed. Users are also reminded that both /expanse/lustre/scratch and /expanse/lustre/projects are not backed up. So please make offsite copies of anything critical in those locations. Thanks SDSC User Services Staff ------------ Dear Expanse User, We are continuing to work on the Expanse Lustre filesystem. The initial problem was due to a hardware issue with one of the metadata server drives. The drive was replaced but problems persist with mounting of storage targets due to a software bug being triggered in the Lustre filesystem. Unfortunately since the metadata server controls the entire filesystem, the /expanse/lustre/scratch and /expanse/lustre/projects directories will continue to be unavailable. We are sorry for the impact this is causing and will keep users posted about any new developments. Thanks SDSC User Services Staff --------- Dear Expanse User, We are continuing to work on the metadata server problem on the Expanse Lustre filesystem. Unfortunately the outage is going to go longer and we will update once we have more information. Thanks SDSC User Services Staff ----------------------------------------- Dear Expanse User, We are continuing to work on the Lustre filesystem on Expanse. The problem is going to take much longer than anticipated to resolve and likely the earliest we can recover is tomorrow (03/18/2025). We recognize that a lot of Expanse users do not use the lustre directories and to enable them to run we will release the reservation. We have held current jobs that are clearly using lustre (e.g. if they specified it in the output path or working directory path). However, if you have jobs that use Lustre without specifying the need through a constraint, the jobs will fail. We strongly recommend all jobs needing Lustre include the following line: #SBATCH --constraint="lustre" Please see more details in our user guide under the "SUBMITTING JOBS USING LUSTRE" subsection in the storage section (https://www.sdsc.edu/systems/expanse/user_guide.html#narrow-wysiwyg-10). We also want to remind users that the home and NFS directories are limited in performance and scaling so please don't submit intensive jobs that need the Lustre filesystem from there. We will keep you posted on the Lustre issue status tomorrow. Thanks SDSC User Services Staff ----------------------------------------

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: tickets.access-ci.org

Start Date: March 17, 2025, 3:30 p.m.

End Date: March 18, 2025, 4:31 a.m.

This may have delayed some of the automation executions up to one day. But, as of now, there's no any other impact identified. https://jira-service-management.status.atlassian.com/incidents/hbhx13fzvdgz#

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: xdmod.access-ci.org

Start Date: March 17, 2025, 2:00 p.m.

End Date: March 18, 2025, 2:00 p.m.

ACCESS XDMoD will be upgraded to version 11.0.1 on Monday, March 17 at approximately 10:00 EDT. Various data in ACCESS XDMoD may be unavailable during the upgrade. Service is expected to be fully restored within 24 hours. Once the upgrade is started, release notes will be available at https://xdmod.access-ci.org/#main_tab_panel:about_xdmod?Release%20Notes

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: registry.access-ci.org

Start Date: March 17, 2025, 1:00 p.m.

End Date: March 17, 2025, 1:30 p.m.

On March 17, 2025, several Enrollment Plugins used by the ACCESS User Registry (https://registry.access-ci.org/) will be updated for compatibility with future versions of COmanage Registry (https://spaces.at.internet2.edu/display/COmanage/Registry+4.4.0+Release+Announcement). Server instances will be restarted during this update which may cause in-progress registrations/logins to fail.

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: faster.tamu.access-ci.org

Start Date: March 10, 2025, 2:00 p.m.

End Date: March 15, 2025, 5:00 p.m.

The FASTER cluster will be unavailable from 9am March 10 to 8pm March 13 for usual OS maintenance, Lustre storage maintenance, and Liqid fabric composability maintenance.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: tickets.access-ci.org

Start Date: March 6, 2025, 5:27 p.m.

End Date: March 7, 2025, 7:00 a.m.

This incident affects: Jira Service Management Web, Service Portal, Opsgenie Incident Flow, Opsgenie Alert Flow, Opsgenie Incident Flow, Opsgenie Alert Flow, Jira Service Management Email Requests, Authentication and User Management, Purchasing & Licensing, Signup, Automation for Jira, and Assist. https://jira-service-management.status.atlassian.com/incidents/hjh7ydq8jlj6

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: registry.access-ci.org

Start Date: March 4, 2025, 4:00 p.m.

End Date: March 4, 2025, 5:00 p.m.

On March 4, 2025, the ACCESS User Registry (https://registry.access-ci.org/) will be reconfigured to include Kubernetes resource requirements. This will enable ACCESS services to respond to user load and create additional server instances when needed. Server instances will be restarted during this update which may cause in-progress registrations/logins to fail. This update has been deployed to the TEST ACCESS User Registry instance at https://registry-test.access-ci.org/ . For questions or concerns with this update, please contact help@cilogon.org (mailto:help@cilogon.org) or open an ACCESS Help Ticket (https://access-ci.atlassian.net/servicedesk/customer/portal/2/create/30).

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org, expanse-ps.sdsc.access-ci.org

Start Date: February 24, 2025, 5:00 p.m.

End Date: July 7, 2025, 4:00 p.m.

Dear Expanse User, Starting Feb 24, 2025, two-factor authentication (2FA) will be required on Expanse. To avoid losing access, set up 2FA with Google Authenticator before this date. Instructions are in the Expanse user guide (https://www.sdsc.edu/systems/expanse/user_guide.html) in the system access section. For role accounts needing programmatic access, Science Gateway PIs will be contacted with further details. Other users with automation needs should contact SDSC user support via ticketing before Feb 24, 2025. SDSC User Services Staff

Posted: March 20, 2026

Infrastructure News Type: Degraded

Affected Infrastructure: duo.access-ci.org

Start Date: February 13, 2025, 6:48 p.m.

End Date: February 14, 2025, 1:43 a.m.

DUO has reported issues with phone-call-based authentication on their systems today, leading to some users being locked out for several hours for too many authentication failures. We do NOT recommend that ACCESS users employ phone-call-based authentication, as it is often unreliable, and prone to being abused. We strongly recommend that ACCESS users configure their DUO authentication to use the DUO App on a mobile device, and to use Push authentication. Other authentication methods, including Passkey or token-based authentication are known to work well. For more information about setting up DUO, please consult the documentation at https://guide.duo.com/universal-prompt#add-or-manage-devices.

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org, expanse-ps.sdsc.access-ci.org

Start Date: February 9, 2025, 6:00 a.m.

End Date: February 9, 2025, 6:00 p.m.

Dear Expanse User, We are currently seeing high load on one of the metadata servers of the Expanse Lustre filesystem. This is leading to timeouts on access to some files and directories. We are looking into the problem and will update once it is resolved. SDSC User Services Staff

Posted: March 20, 2026

Infrastructure News Type: Reconfiguration

Affected Infrastructure: registry.access-ci.org

Start Date: February 6, 2025, 2:00 p.m.

End Date: February 6, 2025, 3:00 p.m.

On February 6, 2025, the ACCESS User Registry (https://registry.access-ci.org/) will be reconfigured as follows. - The text "Terms and Conditions" will be replaced by "Acceptable Use Policy" throughout the user interface. - The DynamoDB Provisioner will be updated so that SSH pubkeys will have the "comment" field appended to the end of the pubkey. This will emulate the behavior of the LDAP Provisioner. Server instances will be restarted during this update which may cause in-progress registrations/logins to fail. This update has been deployed to the TEST ACCESS User Registry instance at https://registry-test.access-ci.org/ . For questions or concerns with this update, please contact help@cilogon.org (mailto:help@cilogon.org) or open an ACCESS Help Ticket (https://access-ci.atlassian.net/servicedesk/customer/portal/2/create/30).

Posted: March 20, 2026

Infrastructure News Type: Outage Full

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org

Start Date: February 6, 2025, 5:00 a.m.

End Date: February 6, 2025, 4:00 p.m.

Dear Expanse User, We will be working on the SDSC machine room power infrastructure from 9PM (PT), Feb 5, 2025 to 8AM (PT) Feb 6, 2025. This will impact power to racks 1-8, 16, and 17 of Expanse. We have placed a maintenance reservation on these racks for the duration of the work. This will impact job wait times as we get closer to the maintenance period but the remaining resources of the system will available through the maintenance and logins, filesystem access, and job submissions are not expected to be impacted. Thanks SDSC User Services Staff

Posted: March 20, 2026

Infrastructure News Type: Outage Partial

Affected Infrastructure: expanse.sdsc.access-ci.org, expanse-gpu.sdsc.access-ci.org

Start Date: February 3, 2025, 1:00 a.m.

End Date: February 3, 2025, 5:00 a.m.

Update - the issue was resolved last night around 9PM (PT) Dear Expanse User We had a scheduler issue this evening that unfortunately led to jobs in the queue being lost. We apologize for the inconvenience this will cause as jobs will have to be resubmitted. We are looking into the issue and will update once the scheduler service is restored. Thanks SDSC User Services Staff

Posted: March 20, 2026