OpenAI has confirmed that the recent data leak in ChatGPT was caused by a vulnerability in the redis-py open-source library.
In addition to being able to view other users' interactions with the AI chatbot, some ChatGPT users also had access to limited personal and billing information that was inadvertently disclosed.
What was the cause of the data leak in ChatGPT?
ChatGPT experienced an outage on March 20, which caused issues with users accessing their conversation history. However, it was discovered that a more serious problem had occurred during this time. OpenAI notified 1.2% of ChatGPT Plus subscribers via email that their billing information may have been inadvertently viewed by another user when clicking on their own 'Manage Subscription' page during a nine-hour window on the same day. The exposed billing information included the subscriber's first and last name, billing address, credit card type, credit card expiration date, and the last four digits of their credit card number. OpenAI confirmed that there was no evidence to suggest that more than one user viewed the information. On a positive note, the leaked chat history only allowed access to the conversation prompts' titles. OpenAI's internal investigation found that the root cause of the data leak was a bug in the Redis client open-source library redis-py.
According to OpenAI, they use Redis to cache user information in their server and Redis Cluster to distribute the load over multiple Redis instances. They also use the redis-py library to interface with Redis from their Python server, which runs with Asyncio. The library maintains a shared pool of connections between the server and the cluster and recycles a connection to be used for another request once done. However, if a request is canceled after the request is pushed onto the incoming queue but before the response is popped from the outgoing queue, the connection becomes corrupted. This bug leads to the next response that's dequeued for an unrelated request receiving data left behind in the connection. OpenAI noted that unfortunately, on that Monday, they made a change to their server that caused a spike in Redis request cancellations. As a result, many connections returned bad data.
Finding a solution
After patching the bug, OpenAI implemented measures to prevent users from accessing data belonging to others and conducted a thorough analysis of their logs to ensure that the issue had been resolved and to identify affected users. They also enhanced the reliability and scalability of their Redis cluster to minimize the occurrence of connection errors during high loads, a sensible move that gave ChatGPT tremendous popularity.
Since its launch two months prior, ChatGPT had garnered an estimated 100 million monthly active users as of January 2023, making it a favorite among both consumers and businesses. However, the latter should exercise caution and subject both ChatGPT and OpenAI to the same third-party risk management procedures that they would with any other application.
In addition to being able to view other users' interactions with the AI chatbot, some ChatGPT users also had access to limited personal and billing information that was inadvertently disclosed.
What was the cause of the data leak in ChatGPT?
ChatGPT experienced an outage on March 20, which caused issues with users accessing their conversation history. However, it was discovered that a more serious problem had occurred during this time. OpenAI notified 1.2% of ChatGPT Plus subscribers via email that their billing information may have been inadvertently viewed by another user when clicking on their own 'Manage Subscription' page during a nine-hour window on the same day. The exposed billing information included the subscriber's first and last name, billing address, credit card type, credit card expiration date, and the last four digits of their credit card number. OpenAI confirmed that there was no evidence to suggest that more than one user viewed the information. On a positive note, the leaked chat history only allowed access to the conversation prompts' titles. OpenAI's internal investigation found that the root cause of the data leak was a bug in the Redis client open-source library redis-py.
According to OpenAI, they use Redis to cache user information in their server and Redis Cluster to distribute the load over multiple Redis instances. They also use the redis-py library to interface with Redis from their Python server, which runs with Asyncio. The library maintains a shared pool of connections between the server and the cluster and recycles a connection to be used for another request once done. However, if a request is canceled after the request is pushed onto the incoming queue but before the response is popped from the outgoing queue, the connection becomes corrupted. This bug leads to the next response that's dequeued for an unrelated request receiving data left behind in the connection. OpenAI noted that unfortunately, on that Monday, they made a change to their server that caused a spike in Redis request cancellations. As a result, many connections returned bad data.
Finding a solution
After patching the bug, OpenAI implemented measures to prevent users from accessing data belonging to others and conducted a thorough analysis of their logs to ensure that the issue had been resolved and to identify affected users. They also enhanced the reliability and scalability of their Redis cluster to minimize the occurrence of connection errors during high loads, a sensible move that gave ChatGPT tremendous popularity.
Since its launch two months prior, ChatGPT had garnered an estimated 100 million monthly active users as of January 2023, making it a favorite among both consumers and businesses. However, the latter should exercise caution and subject both ChatGPT and OpenAI to the same third-party risk management procedures that they would with any other application.