Experience the difference of "Elite".

About the Dec 26th Outage

  • Posted on
  • By

On December 26th, OwnerRez experienced a database corruption issue with our cloud service provider.

We saw the issue right away and started working on it, but there was a domino effect that grew significantly beyond our initial expectations.  Our engineering staff has been recalled from vacation, connecting in from overseas, etc. to diagnose and resolve the problems as quickly as possible.

At present, there are no indications of any hostile activity or attacks, and no reason to believe that any previously gathered data has been lost. We are currently operating on a restored system with up-to-date database backups that occurred at the same time the issue started to develop.

We will be posting information and analysis (here in this same blog post), and holding a public webinar, as soon as we have a more thorough understanding of what went wrong and future preventive actions.

In the meantime, here is a list of common questions that have been coming in.  This list will grow over time as we add more.

Why did it happen? What caused it?

We are still investigating all the details, and there are puzzling issues we still don't fully understand, but what we can say is this...

This past Saturday, as part of a routine update, we upgraded some database servers with a new configuration that we had used extensively for many months.  The configuration was a very routine thing and, again, something that our engineers have used in their own environments (and elsewhere in testing/staging) for a while.  There was no reason to believe the update or new configuration would cause anything out of the ordinary.

From Saturday until Monday, the new configuration started to corrupt data.  We're not sure exactly when or why (this is the part still being investigated) but several database areas began to be unresponsive.  The entire platform started to go down on Monday, and our remediation/failover processes started to kick in, but there was a domino effect, and the remediation/failover processes were affected similarly as the system bounced from up to down and back again.

The investigation into the bad database configuration is ongoing.  By all accounts, it should not have happened.  Our cloud provider has service guarantees with us, so we are working with them on the investigation.

Were others affected, is this a widespread issue?

No, not that we know of.  This is isolated to something we did (updating a database configuration) to our database servers with our cloud provider.  But the obvious question is - what about other businesses that used the same database configuration?

Yes, indeed - we are wondering the same thing.  We purposefully watch to see if software updates by third parties are stable before upgrading our server configurations.  In fact, we purposefully waited on this database configuration (from back in March) because there were some known issues with it affecting servers.  We skipped over it and used a different configuration that was reported to be stable and in use by others, and we used that configuration in test/stage environments for several months before moving forward.  As mentioned above, we are very puzzled by why it happened and by all accounts, it should not have.

(And no, the Southwest Airlines debacle is not related to this. At least, we highly doubt it.)

Was any data lost? Doesn't restoring a backup mean some recent data was lost?

The short answer is: no lost data, but we are still investigating and there may be discrepancies with third parties.

The reason we say "no lost data" is because we have data redundancies in place that save data to multiple places at the same time.  When our primary systems go down, we still have access to other systems.  When we rebuilt and restored the underlying configurations, we took data from secondary systems that were up to date as of the time the issue happened.

After the recovery period finished, we went back and compared the new data with the corrupted data.  There were no discrepancies in terms of missing records.  In other words, no records were showing in the old corrupted version that didn't exist in the new recovered version.

That being said, there are many channels and 3rd parties that we connect with that also send data back and forth and could have "sent" or "received" some data that never got through during the outage.  So a channel or 3rd party might show something different than what OwnerRez does, which means the data is incorrect on one side or another (or both).  Messaging could be another example of this.  Gmail and other messaging services watch volume and throttle accordingly.  When the final recovery effort fully kicked in, a large firehose of queued messaging went out (eg. from you to guests, or from OR to you).  Gmail may have throttled or dropped some of that due to volume concerns on their side.  We have not seen signs of that, but it is possible.

Also, there were intermittent "up" periods, early in the recovering period, when you (or a third party) may have changed data that ultimately did not get saved when we did the final recovery effort.  So there too, data may be missing.

Was any sensitive data (credit card, phone number, PII) lost or taken?

No. This was not the result of a security breach nor any outside influence. Nothing was hacked or stolen.  This was a database configuration issue where a seemingly-insignificant update led to data corruption and then domino'd elsewhere as our fail-over processes started responding.

Don't you have redundancies in place to prevent this from happening?

Yes, many.  Our engineering team spends a lot of time (and on an ongoing regular basis) managing and analyzing our infrastructure.  We have lots of tooling and logging in place that gives us visibility into errors, slow responses, spikes, partner issues, background queues, and much more.  OwnerRez is large and full of many moving parts, and in each of those moving parts, we store data where it can be replicated and archived stably as it grows.

Yesterday, we saw the issue immediately and our fail-over process kicked into gear, but the database configuration affected every remediation effort that swung into place.  We were forced to pause everything completely, get to the root of the problem, and rebuild configurations before turning the firehose back on.

The bad configuration was, itself, something we tested and used extensively for months beforehand, so we are still puzzled by (and investigating) why it happened.  But we made the decision to stop the failover/automation and roll back to a configuration that we thought would make a bigger impact.  We knew it would take a couple of hours, but we believed (and still do) that it was a better trade-off for a return to stability rather than fight corrupt data in rolling cycles for what could be many days.

But we clearly have work to do. We do not want to shift blame or throw up our hands here.  We are committed to being your elite PMS and channel manager and that comes with certain expectations on your part and responsibilities on our part.  We are taking steps to learn from this and incorporate new processes for additional redundancies and better communication.  Some stark problems became very clear over the past 36 hours.

Does Airbnb automatically reactivate property listings, or must that be done manually?

During the outage, you may have received automatic emails from Airbnb stating that your listings were disabled or hidden because your software (ie. OwnerRez) was offline.  Airbnb does this as a safety precaution so that guests are not double-booking your property since Airbnb has no way of confirming with you (ie. via OwnerRez) that everything is okay.

Post-outage, Airbnb should have reactivated any hidden listings automatically, but you should check your listings and not assume this is the case.  When our system started communicating with Airbnb again, we confirmed that they were turning listings back on again by checking with specific listings and seeing that they were visible.  However, do not trust that this is the case across the board.  Check your listings so that you know for sure.  Our helpdesk has reported that some users are still seeing hidden listings.  That could be for specific reasons that don't apply to you, but everyone should still check that their listings are showing and the calendars are open and bookable.  We apologize for the inconvenience this places on you.

If you did not receive any "hidden listings" email from Airbnb, you should be fine.  Airbnb only hides listings if they can't send bookings through, not for other reasons.  However, if you're worried about it, we recommend that you still verify your listings on Airbnb to make sure they are showing and bookable.  It could be that it happened to you, but you didn't see the email or an email was never sent.

During the outage, I disconnected Airbnb. Now what?

During the outage, some users were recommending to each other to go into Airbnb and manually disconnect OwnerRez from the Airbnb side so that they could return to "platform mode" and deal with guests and bookings directly on Airbnb.  If you did this, there is no way to undo it from the Airbnb side.  You need to go into OwnerRez > Settings > API > Airbnb and reconnect the account from the OwnerRez side.  This is safe to do, and your previous connection will reactivate and merge in changes safely.

I did a full resync for Airbnb, but nothing changed.

Triggering a "full resync" for Airbnb does not happen immediately in OwnerRez, and it never has.  The request for the resync is put into the queue, along with many other sync actions, and is pulled out as the queue processes every few seconds.  Typically, in normal conditions, the queue is processed so quickly it can appear to be immediate, but it always follows the queue.

After the outage, the queues were filled with many waiting requests, so everything took time to process.  If you were clicking the "trigger full resync" option on the channel, it was put into the queue behind other things and probably took a while to process.

Post outage, we have been monitoring all of our queues to watch many types of requests process - bookings, channel syncs, messaging, and so on. If you are waiting for a message to be delivered or channel sync to go through, make sure to give it some time to catch up.  If it's been a while (ie. an hour) reach out to helpdesk, and we'll take a look.

Where are my Booking.com bookings? They aren't showing up on my calendar.

Unlike some of our other channel partners, Booking.com does not provide an automatic method to restore lost or missed data, including booking records. Our team does have access to logs and is manually working through that information to find affected accounts and properly update calendars. This should be completed by the end of today, but, note that not all booking information can be restored. However, the calendar data will be correctly entered, preventing any double bookings.

OwnerRez says my Airbnb properties are fully synced and online, but, I still can't see them on Airbnb.

We are seeing a relative handful of this situation - as in, a few dozen properties out of tens of thousands. It appears to be individual properties affected - that is, you might have 10 Airbnb properties in one API-connected account, 9 are working fine, but 1 refuses to come back online no matter what you do. We've tried several things to get those properties working again and haven't had success either - so, we've filed a Critical bug with Airbnb partner support. If you have properties in this condition and have not already reported it to us, either via the Helpdesk or in this thread, please do so - we'll add your information to the Airbnb bug report and track them all together.

**UPDATE** - All but a couple of the offline Airbnb properties have been restored, and those are expected to be fixed by the end of the day.

121 Comments (add yours)

Matthew H
Dec 27, 2022 10:50 AM
Joined Feb, 2019 6 posts

I've had several years of rock solid performance from OwnerRez, so I continue to have great confidence in you.

Ken T
Dec 27, 2022 10:50 AM
OR Team Member Joined Aug, 2019 1403 posts

You definitely *can not* disconnect and reconnect the Vrbo API on the fly - don't attempt that.

Regarding Airbnb, we have seen some reports of clients using this approach successfully, but have not yet confirmed that recovery from that action is smooth so are not recommending that at this time.  It does seem plausible though, and we'll know for sure in a bit.

Ash
Dec 27, 2022 10:52 AM
Joined May, 2022 6 posts

I understand things happen, BUT....

1. Will your staff be able to track "lost reservation data" and insert back into correct places

2. Was credit card data breached, along with telephone numbers and addresses

3. What can we do to back track and check reservations past, present and future to ensure all is good. Reaching out to guests and informing them of issues will not engender sympathy but distrust

4. I trust that you are taking steps to ensure this does not happen again. What steps to a non techie like me are you taking?

Appreciate everything your guys do. Thank You!

Ken T
Dec 27, 2022 10:52 AM
OR Team Member Joined Aug, 2019 1403 posts

@Wayne - It does look like Airbnb automatically reactivated your property listings once service was restored.  Our belief is that this should occur for everyone, so, you should not need to take any manual action to take that happen.

JAC Rentals
Dec 27, 2022 10:53 AM
Joined Sep, 2022 2 posts

Thanks to the tech guys doing their thing.  

Bill and Donna B
Dec 27, 2022 10:53 AM
Joined Nov, 2019 5 posts

-

Jeremy O
Dec 27, 2022 10:54 AM
Joined Aug, 2022 1 post

Losing access to our calendar for that long was very problematic. We had several check-ins with no idea of who would be arriving and how many people they were bringing. We also missed out on multiple bookings. Potential guests called us to inquire about booking and we weren't able to book them or even to tell if our properties were available. The winter holidays are a very busy time for the hospitality industry; we wouldn't be as annoyed if you weren't doubling your rates. 

RNLH
Dec 27, 2022 10:55 AM
Joined Mar, 2022 4 posts

thanks! Better customer service than Southwest!! That is a cluster melt down!

Kylie R
Dec 27, 2022 11:05 AM
Joined Jun, 2021 3 posts

Our listings are all still hidden on Airbnb. Can we do anything to get back up and running?

BlueSky Getwaway
Dec 27, 2022 11:07 AM
Joined Mar, 2022 32 posts

Please consider away that we can easily do a daily, weekly, monthly database backup ourselves for the future. 

Oumou
Dec 27, 2022 11:08 AM
Joined Sep, 2021 1 post
Airbnb
Hello Omou,
We have noticed that the availabilities of one or more of your listings have not been updated. Notably, a high percentage of booking attempts for dates marked as available in your listing's calendar were declined by your software. As a result, your listings will be hidden from travelers until you resolve the issue. You will not receive bookings while listings are hidden.
The hidden ads are as follows:
Apt 1 bedroom-living room-equipped kitchen-wifi fiber s
Please contact your software provider to ensure that your availability calendar on Airbnb remains up to date. We will monitor your ads daily. As soon as your availabilities are updated, they will automatically be visible to travelers again.
Thanks,
The Airbnb team
Evelyn E
Dec 27, 2022 11:09 AM
Joined Mar, 2018 12 posts

What is the best way to back up all our information on a local HD?  Is something in place that we can set up so that we can at least carry on business even if it requires adding the new information to the OR platform after it is back up?  I would really appreciate guidance here.  Thank you.  

Ken T
Dec 27, 2022 11:10 AM
OR Team Member Joined Aug, 2019 1403 posts

There's no need to do a full database backup yourself, but, you can certainly export all your contacts and bookings via several different methods.  We'll likely be adding more convenient or even automated ways to get this done, but for now, the simplest method is the main Bookings screen, List view, Export.  This will include all future bookings, including guest contact information.

Giani C
Dec 27, 2022 11:15 AM
Joined Oct, 2021 1 post

Thank you so much

Adrian M
Dec 27, 2022 11:19 AM
Joined Dec, 2019 2 posts

I lost a lot of bookings from this and I still have several listing that are hidden in AirBnB so it's not possible for guests to find or book them

Maged B
Dec 27, 2022 11:20 AM
Joined Mar, 2016 12 posts

Thanks for your hard work with fixing the issue during the holidays. OR has been more than rock dependable for me for over five years now and the team has always been extraordinary. Things happen sometimes especially when dealing with multiple external factors. Continue the good work. You guys are awesome!

Evelyn E
Dec 27, 2022 11:21 AM
Joined Mar, 2018 12 posts

Thank you for that info.  I appreciate the speed you have been able to solve the issue and get us back up.  It is comforting that you were so ON IT even during the holiday when everyone expected to be off work.  Thank you for your commitment to all of us who depend on you so much.  Blessings for the New Year to each of you and your families.  

OPCD TX
Dec 27, 2022 11:22 AM
Joined Aug, 2019 24 posts

Thanks guys! 

Blake A
Dec 27, 2022 11:24 AM
Joined Nov, 2021 1 post

I have a booking from VRBO that is not showing up on OwnerRez, any ideas on getting it synced? 

Adrian M
Dec 27, 2022 11:25 AM
Joined Dec, 2019 2 posts

stop receiving updates