Experience the difference of "Elite".

About the Dec 26th Outage

  • Posted on
  • By

On December 26th, OwnerRez experienced a database corruption issue with our cloud service provider.

We saw the issue right away and started working on it, but there was a domino effect that grew significantly beyond our initial expectations.  Our engineering staff has been recalled from vacation, connecting in from overseas, etc. to diagnose and resolve the problems as quickly as possible.

At present, there are no indications of any hostile activity or attacks, and no reason to believe that any previously gathered data has been lost. We are currently operating on a restored system with up-to-date database backups that occurred at the same time the issue started to develop.

We will be posting information and analysis (here in this same blog post), and holding a public webinar, as soon as we have a more thorough understanding of what went wrong and future preventive actions.

In the meantime, here is a list of common questions that have been coming in.  This list will grow over time as we add more.

Why did it happen? What caused it?

We are still investigating all the details, and there are puzzling issues we still don't fully understand, but what we can say is this...

This past Saturday, as part of a routine update, we upgraded some database servers with a new configuration that we had used extensively for many months.  The configuration was a very routine thing and, again, something that our engineers have used in their own environments (and elsewhere in testing/staging) for a while.  There was no reason to believe the update or new configuration would cause anything out of the ordinary.

From Saturday until Monday, the new configuration started to corrupt data.  We're not sure exactly when or why (this is the part still being investigated) but several database areas began to be unresponsive.  The entire platform started to go down on Monday, and our remediation/failover processes started to kick in, but there was a domino effect, and the remediation/failover processes were affected similarly as the system bounced from up to down and back again.

The investigation into the bad database configuration is ongoing.  By all accounts, it should not have happened.  Our cloud provider has service guarantees with us, so we are working with them on the investigation.

Were others affected, is this a widespread issue?

No, not that we know of.  This is isolated to something we did (updating a database configuration) to our database servers with our cloud provider.  But the obvious question is - what about other businesses that used the same database configuration?

Yes, indeed - we are wondering the same thing.  We purposefully watch to see if software updates by third parties are stable before upgrading our server configurations.  In fact, we purposefully waited on this database configuration (from back in March) because there were some known issues with it affecting servers.  We skipped over it and used a different configuration that was reported to be stable and in use by others, and we used that configuration in test/stage environments for several months before moving forward.  As mentioned above, we are very puzzled by why it happened and by all accounts, it should not have.

(And no, the Southwest Airlines debacle is not related to this. At least, we highly doubt it.)

Was any data lost? Doesn't restoring a backup mean some recent data was lost?

The short answer is: no lost data, but we are still investigating and there may be discrepancies with third parties.

The reason we say "no lost data" is because we have data redundancies in place that save data to multiple places at the same time.  When our primary systems go down, we still have access to other systems.  When we rebuilt and restored the underlying configurations, we took data from secondary systems that were up to date as of the time the issue happened.

After the recovery period finished, we went back and compared the new data with the corrupted data.  There were no discrepancies in terms of missing records.  In other words, no records were showing in the old corrupted version that didn't exist in the new recovered version.

That being said, there are many channels and 3rd parties that we connect with that also send data back and forth and could have "sent" or "received" some data that never got through during the outage.  So a channel or 3rd party might show something different than what OwnerRez does, which means the data is incorrect on one side or another (or both).  Messaging could be another example of this.  Gmail and other messaging services watch volume and throttle accordingly.  When the final recovery effort fully kicked in, a large firehose of queued messaging went out (eg. from you to guests, or from OR to you).  Gmail may have throttled or dropped some of that due to volume concerns on their side.  We have not seen signs of that, but it is possible.

Also, there were intermittent "up" periods, early in the recovering period, when you (or a third party) may have changed data that ultimately did not get saved when we did the final recovery effort.  So there too, data may be missing.

Was any sensitive data (credit card, phone number, PII) lost or taken?

No. This was not the result of a security breach nor any outside influence. Nothing was hacked or stolen.  This was a database configuration issue where a seemingly-insignificant update led to data corruption and then domino'd elsewhere as our fail-over processes started responding.

Don't you have redundancies in place to prevent this from happening?

Yes, many.  Our engineering team spends a lot of time (and on an ongoing regular basis) managing and analyzing our infrastructure.  We have lots of tooling and logging in place that gives us visibility into errors, slow responses, spikes, partner issues, background queues, and much more.  OwnerRez is large and full of many moving parts, and in each of those moving parts, we store data where it can be replicated and archived stably as it grows.

Yesterday, we saw the issue immediately and our fail-over process kicked into gear, but the database configuration affected every remediation effort that swung into place.  We were forced to pause everything completely, get to the root of the problem, and rebuild configurations before turning the firehose back on.

The bad configuration was, itself, something we tested and used extensively for months beforehand, so we are still puzzled by (and investigating) why it happened.  But we made the decision to stop the failover/automation and roll back to a configuration that we thought would make a bigger impact.  We knew it would take a couple of hours, but we believed (and still do) that it was a better trade-off for a return to stability rather than fight corrupt data in rolling cycles for what could be many days.

But we clearly have work to do. We do not want to shift blame or throw up our hands here.  We are committed to being your elite PMS and channel manager and that comes with certain expectations on your part and responsibilities on our part.  We are taking steps to learn from this and incorporate new processes for additional redundancies and better communication.  Some stark problems became very clear over the past 36 hours.

Does Airbnb automatically reactivate property listings, or must that be done manually?

During the outage, you may have received automatic emails from Airbnb stating that your listings were disabled or hidden because your software (ie. OwnerRez) was offline.  Airbnb does this as a safety precaution so that guests are not double-booking your property since Airbnb has no way of confirming with you (ie. via OwnerRez) that everything is okay.

Post-outage, Airbnb should have reactivated any hidden listings automatically, but you should check your listings and not assume this is the case.  When our system started communicating with Airbnb again, we confirmed that they were turning listings back on again by checking with specific listings and seeing that they were visible.  However, do not trust that this is the case across the board.  Check your listings so that you know for sure.  Our helpdesk has reported that some users are still seeing hidden listings.  That could be for specific reasons that don't apply to you, but everyone should still check that their listings are showing and the calendars are open and bookable.  We apologize for the inconvenience this places on you.

If you did not receive any "hidden listings" email from Airbnb, you should be fine.  Airbnb only hides listings if they can't send bookings through, not for other reasons.  However, if you're worried about it, we recommend that you still verify your listings on Airbnb to make sure they are showing and bookable.  It could be that it happened to you, but you didn't see the email or an email was never sent.

During the outage, I disconnected Airbnb. Now what?

During the outage, some users were recommending to each other to go into Airbnb and manually disconnect OwnerRez from the Airbnb side so that they could return to "platform mode" and deal with guests and bookings directly on Airbnb.  If you did this, there is no way to undo it from the Airbnb side.  You need to go into OwnerRez > Settings > API > Airbnb and reconnect the account from the OwnerRez side.  This is safe to do, and your previous connection will reactivate and merge in changes safely.

I did a full resync for Airbnb, but nothing changed.

Triggering a "full resync" for Airbnb does not happen immediately in OwnerRez, and it never has.  The request for the resync is put into the queue, along with many other sync actions, and is pulled out as the queue processes every few seconds.  Typically, in normal conditions, the queue is processed so quickly it can appear to be immediate, but it always follows the queue.

After the outage, the queues were filled with many waiting requests, so everything took time to process.  If you were clicking the "trigger full resync" option on the channel, it was put into the queue behind other things and probably took a while to process.

Post outage, we have been monitoring all of our queues to watch many types of requests process - bookings, channel syncs, messaging, and so on. If you are waiting for a message to be delivered or channel sync to go through, make sure to give it some time to catch up.  If it's been a while (ie. an hour) reach out to helpdesk, and we'll take a look.

Where are my Booking.com bookings? They aren't showing up on my calendar.

Unlike some of our other channel partners, Booking.com does not provide an automatic method to restore lost or missed data, including booking records. Our team does have access to logs and is manually working through that information to find affected accounts and properly update calendars. This should be completed by the end of today, but, note that not all booking information can be restored. However, the calendar data will be correctly entered, preventing any double bookings.

OwnerRez says my Airbnb properties are fully synced and online, but, I still can't see them on Airbnb.

We are seeing a relative handful of this situation - as in, a few dozen properties out of tens of thousands. It appears to be individual properties affected - that is, you might have 10 Airbnb properties in one API-connected account, 9 are working fine, but 1 refuses to come back online no matter what you do. We've tried several things to get those properties working again and haven't had success either - so, we've filed a Critical bug with Airbnb partner support. If you have properties in this condition and have not already reported it to us, either via the Helpdesk or in this thread, please do so - we'll add your information to the Airbnb bug report and track them all together.

**UPDATE** - All but a couple of the offline Airbnb properties have been restored, and those are expected to be fixed by the end of the day.

121 Comments (add yours)

Miao & Chip W
Dec 27, 2022 11:26 AM
Joined Mar, 2022 5 posts

Thank you for your teams effort in getting everything back up and running.  I am sure they would have rathrr bern celebrating the holidays.

Derrick A
Dec 27, 2022 11:32 AM
Joined Oct, 2021 1 post

API is also throwing error 

Shelley S
Dec 27, 2022 11:35 AM
Joined Mar, 2022 19 posts

I appreciate your communication during this period and understand things happen. I DON'T understand how a secondary back up server could also go down.

Unfortunately the timing resulted in me losing TWO bookings  - two people trying to book yesterday (one through my API integrated VRBO listing, couldn't book since OR was down) and one directly. 

Resulted in lost bookings and revenue for me. Very disappointing.

Mark M
Dec 27, 2022 11:43 AM
Joined Jun, 2020 4 posts

It's certainly frustrating on our end when we need to adjust a security deposit before it gets released but understand bad things happen in the digital world. The question I have is what exactly happened to initiate such a catastrophic outage?

Heather L
Dec 27, 2022 11:50 AM
Joined Sep, 2020 1 post

Grace Vacation Rentals Maui greatly appreciate the attention paid to the issue.  

mtncreekllc
Dec 27, 2022 11:57 AM
Joined Oct, 2019 1 post

Thank you so much for working quickly to resolve the issue!  The only affect I seem to have had is with AirBnB who disabled my listing due to a calendar sync issue.  Other platforms seem to be working just fine and all are sync'd.  

Shelley S
Dec 27, 2022 11:59 AM
Joined Mar, 2022 19 posts

turning off notifications

Austin D
Dec 27, 2022 12:02 PM
Joined Mar, 2021 5 posts

I understand tech issues happen and I do appreciate your candor and explanation here.  That said, we lost three potential bookings and had airbnb suspend several of our listings.  Stuff happens and I'm not here to pile on, but this is pretty concerning.  I look forward to hearing specifically what steps will be made to try and prevent this from happening again.  Meanwhile, many of us lost revenue because of this.  I'm not asking for anything from OR -- again, I do understand stuff happens -- but maybe keep it in mind next time you plan a price increase for hosts.  Kinda hard to ask us to pay more for your services after something like this happens.  

Samantha H
Dec 27, 2022 12:10 PM
Joined Aug, 2022 1 post

Thanks so much for this quick response and the transparency! One of our listings (Bissell House) is also still missing from Airbnb. Any advice on this?

Lisa S
Dec 27, 2022 12:11 PM
Joined Aug, 2019 4 posts

t

Nick F
Dec 27, 2022 12:11 PM
Joined Feb, 2021 4 posts

We had two listings on Airbnb get "hidden" because bookings did not process during the outage. I spoke with Airbnb this morning to "unhide" the listings and they said this is still an issue with OwnerRez?

Ken T
Dec 27, 2022 12:14 PM
OR Team Member Joined Aug, 2019 1403 posts

We are running a full resync with Airbnb which should get all properties back online without any manual action, unless you took manual action to disconnect them yourself previously.  This process is going slower than expected, but is over half done and should be complete by the end of the day.  Please do not attempt to request a manual resync, as that will only delay the process.

Kismet Prop
Dec 27, 2022 12:21 PM
Joined Sep, 2018 23 posts

I'm sure it was rough night. Very unusual for this system to be done.  In fact, I have never experienced it.  Thanks for working through the day and night to get it back up and running!

Mat W
Dec 27, 2022 12:25 PM
Joined Jan, 2022 1 post

My AirBnB listing has been suspended due this issue. I have several inquiries that are unable to book due to this issue. I’m having to submit an appeal with AirBnB to get my property back to an active status. I’m actively losing money because of this issue. 

Disappointed with the lack of proactive information being provided. OwnerRez users are left to find out that their listing have been suspended at no fault of their own? This product should meant to make STR management easier! Not suspend my listing! Any downtime for this product is unacceptable. 

Based on your comments, it sounds like like this was caused not by an outside bad actor or malicious activity, but by an internal change. I’m curious to see the full report, but assuming this issue was self-inflicted, what is OwnerRez doing to make this right? 

Steve K
Dec 27, 2022 12:28 PM
Joined Apr, 2021 4 posts

Still having synch issues with 3rd parties via API.   Airbnb messaging that occurred during outage has not come through.  

Professional Liability and Errors/Omissions policies on behalf of cloud hosting provider and OwnerRez are designed to handle these types of issues.  Plus as hosts and businness owners you have the ability to purchase insurance like Loss of Business policies to aid in these situations.

Considering backup solution in the future it would be best for it not to be in the same data center and cloud hosting provider.  Sometimes a hot backup can be co-located but you run the risk of what happened yesterday when the single point of failure is related to hosting provider.  Then you do not have redundancy real redundancy.  You may want to consider different cloud provider as a hot backup options to ensure true redundancy.  Also you should have a date time stamp on the file you used for restoral to get everyone backup up and running.  It would be helpful to provide that date time stamp with appropriate time zone so that we can go through our systems to help identify gaps in our data and try to complete restoral.

 

Alana A
Dec 27, 2022 12:52 PM
Joined Oct, 2021 6 posts

Our discounts are no longer working, particularly the Snowbird monthly discount.

Hostably
Dec 27, 2022 12:59 PM
Joined Apr, 2022 17 posts

We lost several high-dollar booking opportunities. Many guests were confused during check-in/out as automated instructions were not sent, which could impact reviews. We spent a long time on the phone with support form Airbnb to Vrbo due to outage-related issues. While the fix is appreciated, it resulted in thousands of dollars worth of lost opportunities as booking inquiries / requests ended up booking elsewhere. It will take more hours going through calendars and messages to look for sync errors and unsent communications. This outage and the price increase of 2023 has soured our experience with this software. Please ensure an outage does not happen again by installing better protections and using backup servers.

Fenwick Vacation
Dec 27, 2022 1:02 PM
Joined May, 2018 20 posts

Completely agree with the last posts.  Bad experience for guests and lists revenues.  Now looking through everything making sure things are. It missing.  Now I am wondering if I need to always have a back up.  

Also poor communication from ownerrez on the outage and also what we as owners could do.  Say stop instant booking?

Fatfatmouse
Dec 27, 2022 1:03 PM
Joined Oct, 2020 1 post

Thank you for acting quickly and resolving the issue. When I couldn’t log in, I panicked! Especially for this holiday peak season. But your quick response made me relieved and I had faith on your engineering team. A day later, all up and running - thank you!

Gwen S
Dec 27, 2022 1:06 PM
Joined Sep, 2022 1 post

I'm seeing now how important all of our notifications are that land in the inbox of my business email, truly grateful I was able to find information there for all of our 20+ properties that I needed regarding lease agreements, messages and data I typically go straight to OwnerRez for. The system is well thought out for just such emergencies. Thanks for getting on this as quickly as you did on our second busiest day of the year, fingers crossed December 31 is better for all of us!