Trial Demo
OwnerRez
Staff Reference » Helpdesk » General Info
Updated today

Support Procedures


Guiding principles

  • Answer with a link to a specific relevant doc.
  • Don’t just answer with the doc only, though – take a minute to type out a paragraph pointing out the exact steps for their scenario and then also link to the doc.
  • If no suitable doc, or any possibility of confusion, include screenshots / Loom.
  • Answer with specific links to relevant pages in the client’s account.
  • If discussing anything in the client account that might easily be changed, either by the client or by passage of time, include a screenshot as documentation of what you saw.
  • If you have written 3 duplicate answers (on different tickets) that weren’t references to a support doc, we need one, or to change the process.  Bring this to the attention of CS leadership, with the examples.
  • One touch solve.
  • Educate beyond the simple answer
    • So the client can handle things on their own next time without needing a support request.
    • If they have to make a support request, so they know how to write a better one with full details that is easier to understand by our staff and more likely to be one-touch-solved.
    • Don’t go down rabbit holes and confuse the client. One touch solve doesn't just mean slamming all possible info into the ticket. Sometimes it’s about deleting things that they don't need to know about that would just cause confusion. Users don’t always need to know the detailed why behind something if it would lead to more questions – that stuff can be a side note to internal support if escalated, but often not needed for the user.

Structure

  • Triage - read every new ticket as it comes in and decide on the appropriate action/assignment [Currently operational only intermittently, aspirational as resources permit]
  • Tier 1 - straightforward answers based on docs and/or common sense.  <5 min to determine solution + <5 minutes to craft answer, else skip (juniors) or escalate (seniors)
  • Tier 2 - research required issues, interactions between different systems that may not be fully documented, etc. <15 min to determine solution + <15 minutes to craft answer, else skip (juniors) or escalate (seniors)
  • Tier 3 - dev research required – reading changelog history that is intelligible only to techies, knowing interactions between system processes and what actually happened, checking database/logs, reading code, stepping through a debugger to see what actually happened, exercising partner integrations.
  • Tier 4 - complex dev research – deeper knowledge of system interactions, debugging in prodtest, deeper knowledge of partner integrations. Also watching T3 and T2 queues and error logs to make sure things are running smoothly and notice patterns of errors.
  • Tier 5 - roadmap/strategic decision required to determine proper action. Or on weeks where tier 4 and on-call is a different person, if Tier 4 can’t solve an issue they should escalate to Tier 5.
  • On-Call - system-wide issues that require immediate action, cases that require making raw database updates. Also watching T3 and T2 queues and error logs to make sure things are running smoothly and notice patterns of errors.
    • If there are enough on call issues that T4 is not meeting SLA Green requirements by COB Tuesday, T4 assignee to email CS and Engineering leadership to request an additional engineer on T4 on Wed to clear the backlog.
    • Anyone can escalate a ticket on call if it looks like that level of severity
      • If it’s during work hours, get a verification by your team lead, T2, etc. before escalating on call.
      • If it’s not during work hours, hopefully someone senior will see it in Helpdesk chat.
      • Always mention On-Call issues in Helpdesk chat with the link to the ticket, and Support <> Dev T4 chat if you have access to that.
    • On Call should see a ticket within one hour and note on the ticket that they’ve seen it and are handling it, and go from there
    • If no answer within an hour, authorized folks (Rex, Adria) use the wake me up email to escalate a Pager-duty alert and put it in Support <> Dev T4 chat.
      • This can also be done immediately for major system down events
    • If no answer within 30 minutes after wake me up ping, consult the schedule for who’s on call:
      https://www.ownerrez.com/support/articles/eng-questions
      And call them using the numbers on the Emergency Contacts:
      https://docs.google.com/spreadsheets/d/1H_08OPEb_VkhjDJvZJAFve0xeC8M6Un3MPh8Z305BzM/edit#gid=0

Workflow

  • Triage reads every new ticket and decides on action. Staffed by one T2 person in a rotation.
    See this article
    Possible actions:
    • Trivial/urgent issues easy to address – known macros, 2fa issues, locked account issues, “i have a problem” with no relevant info that allows tracing the booking etc: address immediately and respond directly
    • Patterns of failures or issues developing
      • Escalate to on call if clearly a system outage level event
      • Raise in T4 chat if questionable. Note that the T4 chat isn't checked constantly especially on weekends. It'll be answered eventually but it may be several hours between checks. For urgent questions, escalate to on call.
    • Known to require T2 research – transition comment and escalate T2
    • All others – assign a T1 if there is not a backlog, adjust priority level of relevant tickets if there is a backlog
  • Tier 1 - 2
    • Answer problem if possible
    • Else, escalate (after reviewing ticket details and providing relevant information in an Internal Note)
    • T1 - transition comment if client had the last response
    • T2 - always transition comment
  • Tier 3
    • Every morning do a check for older tickets:
      • If question is on a ticket assigned to a previous T3, skip it. If they don't answer for over one business day, nudge the previous T3 to hit the ticket.
      • If the question hasn't been answered for 2 business days and you won't get to it today, send the user a note that we're still looking into the issue.
    • Research the issue. If it’s taking longer than 20 minutes to make progress on the problem, escalate with your findings. The time limit is to keep you from getting stuck -- if you know you're on the right track, or it takes longer than 20 minutes between tracing the problem, writing an answer to the user, creating a needed fix etc. that's fine -- just don't get stuck in research mode for more than 20 minutes when T4 can likely shortcut research time.
    • Answer problem if possible
    • If a bug is found, create a GH and Known Issue entry and escalate for verification
      • If the fix is straightforward, create a commit and PR for tier 4/on-call to review
    • Else, escalate
    • Non ticket work:
      • Monitor and respond to Dev Phone a Friend forum
        • Answer questions
        • For bug reports, determine if it's a bug. If so, create a GH and loop in T4 to determine if it's hotfix worthy.
      • Debug GVR price accuracy issues and create GH if needed
        • If it's a known issue and we're working on a GH, note it on the GH as an additional example
        • Let T2 know on new issues (dev phone a friend forum) and update KIT
        • If it’s a bad issue requiring delisting of the account, let T2 know to do that
  • Tier 3 for previous weeks
    • If you’ve answered back to the user, you’ll have the ticket assigned to you. If the user comes back with more detail the next week, answer that ticket – you should check tickets assigned to you for responses once at the end of your day if it's not your escalated week.
  • Unrelated question on same ticket
    • For any level of escalation, if the user asks an unrelated question on the same ticket:
      • Use the Split Ticket button on the user's question to split it out to a new ticket.
      • Make sure the new ticket has the escalation and assignment removed (reset to Team:Helpdesk with no agent) so it goes back to the Triage queue.
      • Put an internal note that it was split because it was an unrelated question and update the ticket title to the new issue.
  • Tier 4/on-call

    The top tier is ultimately responsible for overall issue flow and making sure that issues don’t get missed or stuck and are addressed without delay. T4/on call must take ownership of the issue flow and proactively look for problems, not just address issues escalated to them.

    Tier 4 and on-call may be the same person or different people. If different people are on tier 4/on call, they need to work closely together throughout the week to coordinate using the sysops chat and Changing of the Guard punch list.

    • Address any issues that arise
    • For questions that can’t be answered by T4
      • If it’s because there’s a new T4 that needs to ask technical questions of more senior, assign the ticket to the Architecture group (Chris and Joel currently)
      • If it’s because it’s a business question, escalate to T5 with a comment about why
    • Several times a day, skim the T2 and T3 buckets and see if any issues are stuck or a pattern is forming
    • Several times a day, skim the new issues email feed and see if any issues look odd or potentially problematic, a pattern is forming, etc.
    • Once or twice a day, check the error feed and see if any new errors are popping up or forming a pattern.
    • Monitor the Support <> Dev T4 chat and the sysops@ email and respond to any issues
      • If verified urgent, respond of status of hotfix
      • If determined not to be urgent, respond and ask requester to post in Dev Phone a Friend for bug review
    • On day of release, new issues and errors should be checked much more frequently – at least hourly just after the release and tapering off during the day as we get comfort that the release hasn’t introduced new issues.
    • Verify bugs created by T3, evaluate for on call hotfix – is it urgent, is it an isolated fix that’s low risk, etc.
    • If other bugs are noticed that T3 didn’t catch, create a GH or on call fix and Known Issue entry if needed, and de-escalate with info. If the fix is straightforward and low risk, go ahead and do it.
    • If on-call notices small issues not necessarily urgent, go ahead and fix in on-call card, or at least put in tweaks card
    • If on-call notices places that earlier tiers had problems debugging that could be helped with more access or better feature/query
      • If it can be answered by a canned query, write one or assign T3 to write one. This can be enhanced into a feature over time but canned query can often fix things quick.
      • If it needs a UI feature, make a GH and tag it debug-improvements. Chris to review that tag weekly.
    • Keep a running log of everything done in the Changing Of The Guard agenda for next week
      • Record hotfixes in the On-Call GH
      • Known issues changes (in addition to updating known issue tracker)
      • SQL scripts run (should also be committed to hotfix if they affect multiple users)
      • Open loops of any issue that pops up and hasn’t been chased down and known resolved
  • All levels
    • If an issue comes up more than a few times, think about how to automate it – macro, doc, design simplification, code feature, better debug feature etc.
    • If the process wasn't followed (missing transition comment, missing escalated comment etc);
      • Add the Process Tweak tag
      • Enter an internal comment of explanation as to what could have been done better
      • Kick back to other tier
      • For missing transition comment, just answer to minimize time, but still tag and comment about the miss

When escalating upward from T2

  • Check the known issue tracker to see if there’s an ongoing issue and tag the ticket accordingly instead of escalating. Format the tag <known issue number>-<couple word friendly description of issue>.
  • Transition comment to user if the user hasn’t heard from us in a day
  • If lower tier can see more info is needed, request from user while escalating
  • Transition comment to next tier listing the current escalated questions:
    • If an escalated engineer already answered the client once on this ticket and forgot to assign themselves, assign the ticket to the original escalated engineer
    • Write a bold ESCALATED QUESTION 1, ESCALATED QUESTION 2 etc header for each question to make them clear
    • Always make a new comment for each tier when escalating through multiple tiers. If nothing’s changed, T3 could reference the T2 question instead of duplicating, but should still have the ESCALATION QUESTION 1: see T2, etc. headers
    • Include relevant information found during debugging, where you got stuck, etc.
  • If having to re-escalate because the escalated response didn’t fully answer the question, or had jargon that wasn’t understandable, follow the Process Tweak process above

When escalating down

  • If there’s a bug or outage (whether with us or a partner), check the known issue tracker and add (if missing an entry) or mark resolved (if there was an issue and it was resolved). Escalate back down with a comment pointing to the known issue number.
  • If the answer was in a doc that the lower tier could have found, do an internal comment pointing out the doc and expanding on the situation and how the doc is relevant. Tag the ticket with “Missed Doc Review”.
  • Otherwise (no documented answer), answer to the user directly:
    • Assign yourself on the ticket
    • If the response is very specific to the particular question and there’s no info that would have helped a lower tier debug, just answer the user directly and leave for the lower tier to review
    • If the response would be helpful to users, write a comprehensive answer to the user, go to the docs repo, and create an issue with the “Proposed User Doc” tag. Link to the response and mention where you think the article should be located.
    • If the response would be helpful to internal folks only – debugging steps, information about how multiple background processes, partner integrations etc. work, in addition to writing an answer to the user, also write a comprehensive breakdown of the debugging/systems knowledge to the lower tier. Tag the FD issue with “Proposed Internal Doc”.
  • When writing internal notes to the lower tier, provide detailed information about why, not just what if possible — background information that will help tie together the workflow and reasons, and the debugging steps that can be taken to identify the issue if relevant

Review Tags

Rex to review tagged escalate up/down issues weekly and address.

Ticket Statuses

The following ticket statuses are in use in FreshDesk:

  • Open - The default status of a ticket that is waiting for OR staff to do something with it.  Newly created tickets arrive as Open, and, tickets should remain as Open once escalated or sent to a different department so they will be seen.
  • Pending - The ticket has been answered by OR staff and sent to the client, who may or may not respond.  After 3 days, if there is no response, an automatic reminder is sent to the client. If the client responds, the ticket is moved to Open status.  If the client still doesn't respond, the ticket is automatically moved to Resolved status.
  • Resolved - The ticket has been answered by OR staff, and no response from the client is expected (though it's still possible).  No further reminder will be automatically sent to the client.  After a few days in Resolved status, it will be automatically moved to Closed status.
  • Closed - The ticket is considered completed and has been closed.  While closed tickets can be manually reopened by OR staff, they cannot be reopened by the client - if the client replies to a closed ticket, a new ticket is created in Open status.
  • Waiting on Customer - This is similar to Pending, except that a) no reminder is sent to the client and b) when the wait time of 10 days expires, the ticket is returned to Open status rather than Resolved.
  • Hold for X days - similar to "Waiting on Customer" except for the length of time, and, when the time expires and the ticket is returned to Open status, an internal note is automatically placed saying so.
  • Review (then Resolve) - acts the same as Open, but, its purpose is to indicate that, from the client's perspective, the ticket is resolved.  This status is used to indicate to someone internally that they need to review the ticket, and once they've done so, they shuld place it in Resolve status.
  • Circle Back - not currently used.  Instead, use the "Hold" options.

Sending to Engagement

Tickets should be sent to Engagement if any of the following situations apply:

  • We have attempted to explain something to the client twice, and they still aren’t getting it.
  • We have to deliver the client unusually bad news - often this is best done via a friendly phone call, not just an email.
  • The client has a complicated question where, to correctly answer it, we have to know detailed information that they may have a hard time fully explaining on their own.
  • A sales opportunity is or may be involved - e.g. something that could lead to a ProService, a new account opened, etc.
  • The ticket is from a potential new client who has not yet signed up - in that case, send them the macro offering a free trial, and also send to Engagement to give them a call.
  • The client discusses, mentions, or threatens to potentially close their account.
  • The client is pushing back on an invoice, and we have not been able to resolve it via one email.
  • Level 3 security issue:
    https://www.ownerrez.com/support/articles/Authentication-Security 

How to send to Engagement:

  • Optional - If you would like to be notified of the replies, you may first want to set yourself as a Follower on the Ticket, so you can see how it was handled and learn from the Interaction. 
  • Next, in the Zoho main ticket properties set the Ticket Owner to Teams: Engagement
    (Optionally, as shown in this .gif you could also assign to a specific team member of Engagement, but please do so from within the teams picker, not the Agent picker)
    • Note:  If this was an escalated ticket and you are done with the escalation be sure to remove the esclataion as well


Hit Save:  This will properly route the ticket to the Engagement team.

Note: the above process removes the agent assignment, leaving it open for the Engagement team to take over!  This means the previously-assigned agent will no longer be notified of ticket activity.  If you still want these notifications, add yourself as a Follower as outlined in the first optional step

Sending to Product

The Product team has two points of contact that rely on a myriad of processes for tracking purposes. Those two points of contact are the Team Lead (Shawn Harwell) and the Product Coordinator (Bri Fradella). Should you need to touch base with them, your request likely falls within one of the following flows outlined here. 

  • (Internal) Feature Requests from Team Members - should be posted in the Internal Requests forum: https://www.ownerrez.com/forums/internal-requests
    If there is a corresponding Feature Request on the public forums, please include a link to that request when you post your own internally.

  • Bugs Not Actively Involving a Ticket - if the bug is not directly related to a ticket, it should be posted in Dev Phone a Friend forum.

  • Bugs Discovered Through Ticket Handling - bugs that are brought to your attention via a ticket with a user should always be escalated through the general escalation flow (T2+) for research.

General Questions or Feedback for Product - If the ticket contains ongoing work for the regular support team, but you need to give Product a heads up:

  • In an internal comment, explain why Product needs to be aware of it. Your note should start with "Product" in bold so that it stands out from the rest of the ticket chatter. Do not CC a member of the Product team, rather...
  • Check the Notify Product box in the Ticket Information section of the left panel in the Zoho ticket.
    This checkbox should not be used in the event of bug reporting or in an attempt to suggest a feature. We have other processes for those, as outlined above.
    • Product is not an escalation queue and should not be treated as such
    • T1 should notify Product via Zoho only for general feature feedback
    • All other matters should be routed through Dev first for technical assessment

Forum Post Routing Guidelines

General Help Forums Stays with Support.

Feature Requests Follow this decision tree in order:

  1. Is this a duplicate request? (You can use the macro /cai in Claude and paste the request URL if you need some help)
    • Merge with the existing request.
  2. Does the post include a bug report?
    • Escalate to T3. No transition note needed for the user.
      • If there's no further detail required from the client involving sensitive account information, just escalate the forum ticket to T3.
      • If additional information is needed that should not be discussed in the forum, create a Zoho ticket and reach out to that user directly. Escalate as needed once details are gathered.
    • T3 should ensure that the forum thread is linked on the GH bug report that will then populate in the DPAF forum as a Known Issue.
    • T2 to follow up in the forum thread once a bug is confirmed.
    • T2 to close the Feature Request if it's determined to be a bug.
  3. Is there a workaround or existing solution?
    • Solution / Already Possible? Provide the solution to the user and set the thread status to Closed.
    • If the workaround is overly complex or reveals a usability gap:
      • Post one of the /fr1 macro replies in the thread.
      • Forward to Product team using the Notify Product checkbox.
  4. If none of the above apply:
    • Post one of the /fr1 macro replies in the thread.
    • Forward to Product team using the Notify Product checkbox.

Reporting Support Article Issues

Reporting support article issues is imperative to the success of the Support Center. Here are the instructions on how to report support article issues.

Sending to Marketing

Paul Hall is head of Marketing and Partnerships. Any of the following needs to be brought to his attention:

  • Request from someone that looks like a potential new partner, e.g. wanting to use one of our APIs to provide services to our clients.
  • A systemic or significant problem with a partner - that is, not just one specific client having problems, but ongoing technical issues, missing features, etc.
  • Lack of response from a partner to a client reaching out to them.
  • Requests for T-shirts or other merch.

How is this done?

  • If the ticket contains ongoing work for the regular support team:
    • In an internal comment, explain why Paul Hall needs to be aware of it.  Make sure to @Paul Hall in the comment - you'll see him automatically added as a CC.
    • Add him as a Watcher
  • If the ticket is otherwise finished and ready to be Resolved:
    • In an internal comment, explain why Paul Hall needs to be aware of it.
    • Assign it to Marketing / Paul Hall.

Forum ticket feature requests

  • Often, the feature request is already possible to do, no engineering work required.  If so, senior CS staff / T2 reply accordingly and educate the user on the forum, then close the ticket.
  • For feature requests that are legitimate and seem reasonable, in an internal comment, explain why Product needs to be aware of it. Your note should start with "Product" in bold so that it stands out from the rest of the ticket chatter. Do not CC a member of the Product team, but do check the Notify Product box in the Ticket Information section of the left panel in the Zoho ticket.
  • For feature requests that seem unnecessary and have low votes, just close the ticket.
  • Do not escalate feature request forum posts to T3/4.

Opportunities for Improvement (OFI)

Sometimes, when working a ticket, we identify something that's "not quite right" that could be fixed by Engineering, but, isn't noticeable enough or clear enough to clients that it would be best handled via a Forum feature request.

  1. Make sure that the actual content of the ticket (from the client's perspective) has been fully handled, and would otherwise be Resolved.
  2. Document the OFI so it can be clearly understood.
  3. Your note should start with "Product" in bold so that it stands out from the rest of the ticket chatter. Do not CC a member of the Product team, but do check the Notify Product box in the Ticket Information section of the left panel in the Zoho ticket.

Known issues

  • If known issue exists, tag ticket, add to tracker/master ticket that’s already escalated
  • Escalation down will add/resolve if it is a known issue
  • On-call will update more info on issue as the status changes
  • Review known issues for week at Changing Of The Guard meeting
  • T3/T4 will add the known issue tracker item and GH if applicable (either individual GH if not urgent or on-call fix)
    • Write the bug title/on call fix entry in the format used by the changelog (product to provide guidelines)
    • If T3 creates a bug, escalate to T4 rather than T2 so that T4 can decide if it’s an on-call fix or will go into the normal bug flow which may take awhile to fix.

Partner questions

  • For channel partner procedures, See also this doc
  • For other partners, find the contact in the Partner Contacts doc and email them – use the YELLOW only, if there is none, escalate.
  • If there’s a known procedure that requires a partner ticket, mention that to the previous tier and let them open and manage the ticket.
  • If an issue involving a partner ticket involves multiple users, it needs to be on the Known Issue tracker too.
  • If the escalated research developed into the need for a partner ticket, open the ticket directly (don’t de-escalate and ask them to file the ticket). If you are absolutely certain that the lower tier can handle any response from the partner then de-escalate after opening the ticket, otherwise leave it in the current tier.
  • If email, open a side thread on the ticket. Ask for the partner ticket number and enter into Known Issues tracker if applicable.
  • If ticket form, after opening the ticket with the partner, put an internal note that the partner ticket was opened with the ticket number and link. Enter into Known Issues tracker if applicable.
  • CC dev@ownerrez.com and tier2@ownerrez.com on all partner communications of any type.

Changing of the Guard

  • Weekly meeting on Monday after release planning to keep everybody in sync
  • Joined by T3 and On-Call designee for this week, T3 and On-Call designee from previous week, T2+ support
  • Cover what happened last week, answer any questions, update Known Issue tracker
  • Review changes going out this week
  • Review any open issues not finished from On-Call last week and either close or assign them out for research this week

Service Level Agreement (SLA) Response Times

Time counted from the ticket’s first appearance in the specified view

 

Green

Yellow

Red

Triage/T1

< 2 business hours

< 1 business day

> 1 business day

T2

< 1 business day

< 2 business days

> 2 business days

T3-Eng1

< 1 business day

< 2 business days

> 2 business days

T4-Eng2

< 2 business days

< 3 business days

> 3 business days

T5-Arch

< 2 business days

< 3 business days

> 3 business days

 

Cross week tickets

  • At the end of T3 shift, leave any unfinished tickets where you haven’t responded to the user or previous tier open and provide any info you’ve already researched for the next T3
  • If you did respond to the user, the ticket will be assigned to yourself, so if there is followup the next week, handle it
  • T4 shift will be covered by On-Call doc and Changing of the Guard

Ongoing open issue with many tickets

  • How many tickets do we see before opening a Sorry (status page)?
    • 6 as a general guide, based on severity and visibility
  • When do we create a tag? 
    • Issues needing investigation with 3+ tickets as a general guide

Use of Sorry (OwnerRez Status page)

  • Who?
    • Sysops Team
    • Chris
    • PW
    • Shawn
    • Rex
  • What constitutes an issue sufficiently grave to be posted?
    • Investigating…  Anything that triggered an On-Call.  Investigations aren’t blasted.
    • Known Issue… Blasted IF it’s affecting a large percent of clients – thousands, potentially many hundreds if it’s a large or important issue
  • Process
    • Normal work hours: Helpdesk and T4 consult in Support <> Dev T4 chat to decide 
    • Outside work hours: Anyone with authority to make a post on their best judgment
      • Only for issues sufficiently critical to trigger On-Call.  But, if the problem is widespread, there is no need to wait for an On-Call response or analysis