title: SVN Service Outage - PostMortem
date: '2014-12-09T09:58:37+00:00'
permalink: svn_service_outage_postmortem
layout: post

<p> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>Summary</strong></span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"></span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;">On Wednesday December 3rd the main US host for the ASF subversion service fails resulting in loss of service. &nbsp;This loss of subversion service prevent committers from submitting any changes, and whilst we have an EU mirror it is read-only and does not allow for any changes to be submitted whilst the master is offline.</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"></span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;">The cause of the outage was a failed disk. This failed disk was part of a mirrored OS pair. &nbsp;Some time prior to this the alternate disk had also been replaced due to a failed state.</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"></span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>Timeline</strong></span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"></span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>0401 UTC 2014-10-26 -</strong> eris daily run output notes the degraded state of root disk gmirror</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>1212 UTC 2014-10-30 -</strong> INFRA-8551 created to deal with gmirror degradation.</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>2243 UTC 2014-12-02 -</strong> OSUOSL replaced disk in eris</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>0208 UTC 2013-12-03 -</strong> Subversion begins to crawl to a halt</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>0756 UTC 2013-12-03 -</strong> First contractor discovers something awry with subversion service</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>0834 UTC 2013-12-03 -</strong> Infrastructure sends out a notice about the svn issue</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>0916 UTC 2013-12-03 -</strong> Response to issue begins</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>1010 UTC 2013-12-03 -</strong> First complaints about mail being slow/down</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>1025 UTC 2013-12-03 -</strong> Discovery that email queue alerts had been silenced.</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>1225 UTC 2013-12-03 -</strong> Discovery that Eris outage affecting LDAP-based services including Jenkins and mail</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>1613 UTC 2013-12-03 -</strong> First attempt at power cycling eris</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>1717 UTC 2013-12-03 -</strong> Concern emerges that the 'good' disk in the mirror isn't.</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>1744 UTC 2013-12-03 -</strong> OSUOSL staff shows up in the office</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>1752 UTC 2013-12-03 -</strong> Blog post went up.</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>1906 UTC 2014-12-03 -</strong> New hermes/baldr (hades) being set up for replacement of eris</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>1911 UTC 2014-12-03 -</strong> #svnoutage clean room in hipchat began</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>2040 UTC 2014-12-03 -</strong> machine finally comes up and is usable.</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>2050 UTC 2014-12-03 -</strong> confusion arises between which switch is in which rack. Impedance mismatch between what OSUOSL calls racks, and what we called racks.</span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; [Dec-3 5:50 PM] Tony Stevenson: which rack is this<br /></span><span style="font-family: Arial; -webkit-text-stroke-color: #000000;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; [Dec-3 5:50 PM] Tony Stevenson: 1, 2 or 3 <br />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; [Dec-3 5:50 PM] Justin Dugger (pwnguin): 19&nbsp; <br />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; [Dec-3 5:50 PM] David Nalley: what switch? <br />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; [Dec-3 5:50 PM] Justin Dugger (pwnguin): HW type: HP&nbsp; &nbsp; &nbsp; ProCurve 2530-48G&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; OEM S/N 1: CN2BFPG1F5 <br />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; [Dec-3 5:50 PM] David Nalley: ^^^^^^^^^ points to this impedance mismatch for the postmortem <br />&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; [Dec-3 5:50 PM] David Nalley: no label on the switch?<br /></span><strong style="font-family: Arial; -webkit-text-stroke-color: #000000;">2054 UTC 2014-12-03 -</strong><span style="font-family: Arial; -webkit-text-stroke-color: #000000;"> Data copy begins<br /></span><strong style="font-family: Arial; -webkit-text-stroke-color: #000000;">0441 UTC 2014-12-04 -</strong><span style="font-family: Arial; -webkit-text-stroke-color: #000000;"> data migration finished<br /></span><strong style="font-family: Arial; -webkit-text-stroke-color: #000000;">1457 UTC 2014-12-04 -</strong><span style="font-family: Arial; -webkit-text-stroke-color: #000000;"> SVN starts working again - testing begins<br /></span><strong style="font-family: Arial; -webkit-text-stroke-color: #000000;">0647 UTC 2014-12-05 -</strong><span style="font-family: Arial; -webkit-text-stroke-color: #000000;"> svn-master is operational again with viewvc</span></p> 
  <p><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"></span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>Problems</strong></span><br /> </p> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">It took us far too long to spin up replacement machine. This in fact took a few hours due to having to manually build the host from source media and encountering several BIOS/RaidController issues. &nbsp;Our endeavour to have automated provisioning of tin (bare metal) would certainly have improved this time considerably had it been available at the time of the event. &nbsp;</span></li> 
  </ul> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">Many machines pointing to eris.a.o for LDAP - not to a service name (such as ldap1-us-west for example) which meant we couldn’t easily restore LDAP services for some US hosts without making them also think SVN services had also moved.&nbsp;</span></li> 
  </ul> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">Assigning of issues in JIRA - It has perhaps been a long held understanding that if an issue is assigned to someone in JIRA then they are actively managing that issue. This event clearly shows how fragile that belief is.</span></li> 
  </ul> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">DNS (geo) updates were problematic - Daniel will be posting a proposal on Thursday, which will outline our concerns around DNS and a viable way forward that meets our needs and is not reliant on us storing all the data in SVN to be able to effect changes to zones. (This proposal was not created as a tiger of this event, it has been worked on for a number of weeks now).</span></li> 
  </ul> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">architectural problems for availability</span></li> 
  </ul> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">We couldn't promote svn-eu to master - data differences/corruption&nbsp;<span style="font-kerning: none; color: #042eee; -webkit-text-stroke-color: #042eee;"><u><a href="https://issues.apache.org/jira/browse/INFRA-6236">https://issues.apache.org/jira/browse/INFRA-6236</a><br /><br /></u></span></span></li> 
    <li>Current monitoring setup was not sufficient in catching disk errors and correctly alerting infra.&nbsp;</li> 
    <p> </p> 
  </ul> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"></span><br /> <span style="font-stretch: normal; font-size: 14px; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>To Do</strong></span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"></span><br /> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">Daniel to investigate and evaluate multimaster service availability.</span></li> 
  </ul> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">Implement an extended SSL check that not only ensures the service is up, but also checks cert validity (expire, revocation status etc), and the certificate chain is valid.</span></li> 
  </ul> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">De-couple DNS from SVN</span></li> 
  </ul> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">De-couple the SVN authz file from SVN directly. Also breser@ has suggested we use the authz validation tool available from the svn install we have on hades, &nbsp;as part of the template-&gt;active file generation process.</span></li> 
  </ul> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">Move the ASF status page (http://status.apache.org) outside of our main colos so folks can continue to see it in the event of an outage.</span></li> 
  </ul> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">Vendor provided hardware monitoring tools mandatory on all new hardware deployments.</span></li> 
  </ul> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">Broader audience for incidents and status reports</span></li> 
  </ul> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">More aggressive host replacement before these issues arise&nbsp;</span></li> 
  </ul> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"></span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"></span><br /> <span style="font-stretch: normal; font-size: 14px; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>Things being considered</strong></span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"></span><br /> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">Mandatory use of SNMP for enhanced data gathering.&nbsp;</span></li> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">Issue ‘nagging’ - develop some thoughts and ideas around the concept of auto-transitioning un-modified JIRA issues after N hours of in activity and actively nag the group until an update is made. This for example is how Atlassian (and so many others) handle their issues. &nbsp;For example if an end-user doesn’t update the issue within 5 days, it is automatically closed, if we don’t update an open issue within 6 hours for a critical issue then we get nagged about it.&nbsp;</span></li> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">Automatically create new JIRA issues (utilising above mentioned auto-transition) to notify of hardware issues (not just relying on hundreds of cron emails a day).</span></li> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">Again as part of a wider thinking of how we use issue tracking consider the concept that you only assign an issue to yourself if you are explicitly working on it at that moment, i.e it should not sit in the queue assigned to someone for &gt; N hours and not receive any updates.&nbsp;</span></li> 
  </ul> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"></span><br /> <span style="font-stretch: normal; font-size: 14px; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"><strong>Things that went well</strong></span><br /> <span style="font-stretch: normal; font-family: Arial; font-kerning: none; -webkit-text-stroke-color: #000000;"></span><br /> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">The people working on the issue worked extremely well as a team. &nbsp;Communicating with one another via hipchat and helping each other along where required. &nbsp;There was a real sense of camaraderie for the first time in a very long time and this see of team helped greatly.&nbsp;</span></li> 
  </ul> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">The team put in a bloody hard shift.</span></li> 
  </ul> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">There is now a very solid understanding of the SVN service across at least 4 members of the team, as opposed to 2 x0.5 understandings before.</span></li> 
  </ul> 
  <ul> 
    <li style="margin: 0px; font-stretch: normal; font-family: Arial; -webkit-text-stroke-color: #000000; -webkit-text-stroke-width: initial;"><span style="font-kerning: none;">A much broader insight into the current design of our infrastructure was gained by the newer members of the team.&nbsp;</span></li> 
  </ul>
