Update - Following our latest notice on Saturday April 13th, 2024, we are writing to provide our Customers with an update on the latest available information regarding the Sisense Data Breach.
On Tuesday April 16, 2024, BigPanda was informed by Sisense that the Breach involved Sisense data cubes.
With this new information, we are finalizing a process to export the data that was compromised in the Sisense cubes and provide it to our Customers. We expect this process to be complete by the end of the week, at which time we will be able to schedule individual times upon request.
Our investigation is still ongoing. For any additional requests or to arrange further discussion, please contact the BigPanda Trust & Security team by email at trust@bigpanda.io.
Apr 18, 2024 - 07:20 CEST
On Tuesday April 16, 2024, BigPanda was informed by Sisense that the Breach involved Sisense data cubes.
With this new information, we are finalizing a process to export the data that was compromised in the Sisense cubes and provide it to our Customers. We expect this process to be complete by the end of the week, at which time we will be able to schedule individual times upon request.
Our investigation is still ongoing. For any additional requests or to arrange further discussion, please contact the BigPanda Trust & Security team by email at trust@bigpanda.io.
Apr 18, 2024 - 07:20 CEST
Update - BigPanda became aware of a Data Breach through its subprocessor Sisense on Thursday April 11th, 2024. We informed Customers that same day of the Breach via the BigPanda Status Page (status.bigpanda.io). We are writing now to notify our Customers and provide an update on the latest available information.
Incident Summary
A Threat Actor compromised the Sisense platform and gained access to BigPanda Customer Data through Snowflake. To our knowledge, the situation has been contained. Our investigation indicates that at no time did the Threat Actor gain access to BigPanda’s Core Platform.
BigPanda’s Use of Sisense and Snowflake
The BigPanda Core Platform stores metrics data in Snowflake, which is then loaded into Sisense to deliver metrics reporting via Unified Analytics.
Incident and Data Scope
Sisense has confirmed the Breach but has yet to provide detailed information about the incident.
After BigPanda’s internal investigation, it was confirmed that the Threat Actor accessed BigPanda Customer Data stored in Snowflake via Sisense. It was determined that a read-only service account credential for Snowflake was compromised due to the Breach, thus enabling the threat actor to make SQL queries on the Snowflake database.
The BigPanda Customer Data accessed included usernames (which contain BigPanda customer email addresses) and alert and incident tag information. This data in Snowflake enables customers to identify trends within their environments, examples are included below:
Username (email address), app_id, source_host, priority, start_time, end_time, date, etc
Incident Containment & Remediation
Upon becoming aware of the incident, the BigPanda Security Incident Response Team immediately launched an investigation, which included execution of the following steps:
- Requesting information and contacting Sisense;
- Reviewing all internal systems audit logs for abnormal activity;
- Performing programmatic updates to rotate user access keys and BigPanda’s SSO token via prepared scripts;
- Rotating service account keys for systems that feed data to Sisense cubes.
- Stopping any new Customer Data from being sent to Sisense. We will re-evaluate this decision in the days to come based on information that will be provided to us from Sisense.
A more detailed list of the actions taken by the BigPanda Security Incident Response team to ensure the containment of the Breach are as follows:
- Changed all Sisense-related passwords on my.sisense.com
- For non-Single Sign-On (SSO) access:
-- Replaced the Secret in the Base Configuration Security section with your GUID/UUID.
-- Reset passwords for all users in the Sisense application.
-- Logged out all users by running GET /api/v1/authentication/logout_all under Admin user.
- For SSO access:
-- Updated sso.shared_secret in Sisense and updated the newly generated value in the SSO handler.
-- Rotated the x.509 certificate in our SSO SAML identity provider.
-- Rotated the OpenID client secret.
-- Updated SSO settings in Sisense with the revised values.
-- Logged out all users by running GET /api/v1/authentication/logout_all under Admin user.
-- Reset credentials in the database used by the Sisense application.
Ongoing Investigation and Mitigation
BigPanda is continuing to investigate the issue and will conduct a full external forensic investigation of its Data Platform. We will continue to provide updates as they become available.
For any additional requests or to arrange further discussion, please contact the BigPanda Trust & Security team by email at trust@bigpanda.io.
Apr 13, 2024 - 22:14 CEST
Incident Summary
A Threat Actor compromised the Sisense platform and gained access to BigPanda Customer Data through Snowflake. To our knowledge, the situation has been contained. Our investigation indicates that at no time did the Threat Actor gain access to BigPanda’s Core Platform.
BigPanda’s Use of Sisense and Snowflake
The BigPanda Core Platform stores metrics data in Snowflake, which is then loaded into Sisense to deliver metrics reporting via Unified Analytics.
Incident and Data Scope
Sisense has confirmed the Breach but has yet to provide detailed information about the incident.
After BigPanda’s internal investigation, it was confirmed that the Threat Actor accessed BigPanda Customer Data stored in Snowflake via Sisense. It was determined that a read-only service account credential for Snowflake was compromised due to the Breach, thus enabling the threat actor to make SQL queries on the Snowflake database.
The BigPanda Customer Data accessed included usernames (which contain BigPanda customer email addresses) and alert and incident tag information. This data in Snowflake enables customers to identify trends within their environments, examples are included below:
Username (email address), app_id, source_host, priority, start_time, end_time, date, etc
Incident Containment & Remediation
Upon becoming aware of the incident, the BigPanda Security Incident Response Team immediately launched an investigation, which included execution of the following steps:
- Requesting information and contacting Sisense;
- Reviewing all internal systems audit logs for abnormal activity;
- Performing programmatic updates to rotate user access keys and BigPanda’s SSO token via prepared scripts;
- Rotating service account keys for systems that feed data to Sisense cubes.
- Stopping any new Customer Data from being sent to Sisense. We will re-evaluate this decision in the days to come based on information that will be provided to us from Sisense.
A more detailed list of the actions taken by the BigPanda Security Incident Response team to ensure the containment of the Breach are as follows:
- Changed all Sisense-related passwords on my.sisense.com
- For non-Single Sign-On (SSO) access:
-- Replaced the Secret in the Base Configuration Security section with your GUID/UUID.
-- Reset passwords for all users in the Sisense application.
-- Logged out all users by running GET /api/v1/authentication/logout_all under Admin user.
- For SSO access:
-- Updated sso.shared_secret in Sisense and updated the newly generated value in the SSO handler.
-- Rotated the x.509 certificate in our SSO SAML identity provider.
-- Rotated the OpenID client secret.
-- Updated SSO settings in Sisense with the revised values.
-- Logged out all users by running GET /api/v1/authentication/logout_all under Admin user.
-- Reset credentials in the database used by the Sisense application.
Ongoing Investigation and Mitigation
BigPanda is continuing to investigate the issue and will conduct a full external forensic investigation of its Data Platform. We will continue to provide updates as they become available.
For any additional requests or to arrange further discussion, please contact the BigPanda Trust & Security team by email at trust@bigpanda.io.
Apr 13, 2024 - 22:14 CEST
Monitoring - We want to inform you that we have identified and stopped some suspicious activities from an unidentified threat actor. Pursuing our audit of this security incident, we were able to identify there was read-only access to some user data in the US. No data was extracted from the EU database.
This incident follows a security incident experienced by one of our partners, Sisense. We wanted to let you know that BigPanda takes this matter extremely seriously. We have started a detailed investigation immediately to understand the full extent of what is happening. Our internal investigation determined the data compromise did not include operational data and was limited to analytical data used for our Unified Analytics offering.
As an immediate remediation, we updated all the user access keys and credentials for all our systems. This has terminated access for the threat actor and prevents further intrusion from the recent exploit. We monitored and confirmed there has been no threat actor activity after the user access keys were changed.
The threat actor was able to query a list of usernames, which are email addresses. There is no indication that customer API Keys were compromised since all API Keys are stored in a separate secured environment, which was not compromised as part of this event.
All user access credentials were rotated as of April 11th, 1:45 PM Pacific, and we have completed Sisense recommended procedures.
For any additional requests or to arrange further discussion, please contact BigPanda Security team by email trust@bigpanda.io.
Apr 12, 2024 - 09:11 CEST
This incident follows a security incident experienced by one of our partners, Sisense. We wanted to let you know that BigPanda takes this matter extremely seriously. We have started a detailed investigation immediately to understand the full extent of what is happening. Our internal investigation determined the data compromise did not include operational data and was limited to analytical data used for our Unified Analytics offering.
As an immediate remediation, we updated all the user access keys and credentials for all our systems. This has terminated access for the threat actor and prevents further intrusion from the recent exploit. We monitored and confirmed there has been no threat actor activity after the user access keys were changed.
The threat actor was able to query a list of usernames, which are email addresses. There is no indication that customer API Keys were compromised since all API Keys are stored in a separate secured environment, which was not compromised as part of this event.
All user access credentials were rotated as of April 11th, 1:45 PM Pacific, and we have completed Sisense recommended procedures.
For any additional requests or to arrange further discussion, please contact BigPanda Security team by email trust@bigpanda.io.
Apr 12, 2024 - 09:11 CEST
Inbound Integrations and Event Processing
Operational
Data Consumption
Operational
Alert Enrichment
Operational
Alert Filtering & Maintenance Plans
Operational
Incident Enrichment & Environments Population
Operational
Correlation
Operational
Outbound Collaboration
Operational
Incident Sharing
Operational
Outbound Integrations
Operational
ETL Pipeline
Operational
Console Functions
Operational
Login
Operational
Incident Feed
Operational
Incident Feed Search
Operational
Automatic Incident Triage
Operational
Root Cause Changes
Operational
Administrator Screens
Operational
Incident Actions
Operational
Incident Activity Feed
Operational
Unified Search
Operational
Unified Analytics
Operational
Integration Diagnostics / Troubleshooting
Operational
APIs
Operational
Tier 1 APIs: Automation & Event Processing
Operational
Tier 2 APIs: Supplemental
Operational
Tier 3 APIs: Tertiary
Operational
Third Party
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
Restore per-region buckets to OIM
May 20, 2024 15:00-17:00 CEST
In support of our commitment towards improving our services and infrastructure, we will be performing the following maintenance to the BigPanda platform.
Start Time: May 20, 2024 15:00 UTC
End Time: May 20, 2024 17:00 UTC
Duration: 2 hours
What's Happening?
During this maintenance window, we will be performing an update to our OIM services by migrating back to using per-region S3 buckets for the OIM service’s high availability clusters.
Why is it Happening?
The current process could create a single point of failure for some environments in the unlikely event that S3 experiences a full outage in the operating regions. To mitigate this low risk, we are working to restore the previous functionality of operating out of a single bucket per region of operation.
Component(s) undergoing maintenance:
• Pipeline
What's the Impact?
This change affects the OIM pipeline in all environments. We have done extensive preparation and taken every precaution to ensure a fully smooth transition.
Users may experience up to 15 minutes of processing latency each on alerts sent to any OIM-based integration as the changes are deployed.
What isn't Impacted?
All other areas of the BigPanda platform - including data ingest and processing, APIs, Incident Sharing, and Analytics - will be unaffected by this maintenance.
Is User Action Required?
No user action is required at this time.
We will be closely monitoring the maintenance work, and BigPanda's status page will be updated throughout the maintenance period with any changes to service operations.
We apologize for any inconvenience that this may cause. If you have any questions, please reach out to BigPanda Support via our live in-app chat or via support@bigpanda.io.
Posted on May 18, 2024 - 00:06 CEST
Start Time: May 20, 2024 15:00 UTC
End Time: May 20, 2024 17:00 UTC
Duration: 2 hours
What's Happening?
During this maintenance window, we will be performing an update to our OIM services by migrating back to using per-region S3 buckets for the OIM service’s high availability clusters.
Why is it Happening?
The current process could create a single point of failure for some environments in the unlikely event that S3 experiences a full outage in the operating regions. To mitigate this low risk, we are working to restore the previous functionality of operating out of a single bucket per region of operation.
Component(s) undergoing maintenance:
• Pipeline
What's the Impact?
This change affects the OIM pipeline in all environments. We have done extensive preparation and taken every precaution to ensure a fully smooth transition.
Users may experience up to 15 minutes of processing latency each on alerts sent to any OIM-based integration as the changes are deployed.
What isn't Impacted?
All other areas of the BigPanda platform - including data ingest and processing, APIs, Incident Sharing, and Analytics - will be unaffected by this maintenance.
Is User Action Required?
No user action is required at this time.
We will be closely monitoring the maintenance work, and BigPanda's status page will be updated throughout the maintenance period with any changes to service operations.
We apologize for any inconvenience that this may cause. If you have any questions, please reach out to BigPanda Support via our live in-app chat or via support@bigpanda.io.
Posted on May 18, 2024 - 00:06 CEST