How Heart Internet web hosting almost destroyed my business

If you’re a regular reader of any of my websites,  you’ve probably already noticed that all of the sites have been down for most of today.

You’ve probably also noticed that, as of the time of writing (8pm, Friday), although most of the sites are now back up, they’re all displaying posts from September, with over a month’s worth of content missing.

This is because, at about 2pm today, our web hosts, Heart Internet, called us to say that while they were working on our server (our sites are huge and take up a lot of space, so we have our own server with them), they had somehow managed to irretrievably lose all of the sites. As in, they were gone forever, with no way of restoring them.

Our server was mirrored – there were two discs, each containing the same information. They had both suffered the same catastrophic event (we still don’t know what that was, exactly), and the data on both of them was lost. Forever.

Just to be clear what we’re talking about here, they told us they had lost this site, The Fashion Police (over 600,000 page impressions per month), Hey, Dollface! (around 90,000 per month) and  Shoeperwoman (70,000 page impressions per month), PLUS all of the websites Terry has designed and hosts for our clients. Gone. All gone.

I realise a lot of people don’t take blogging seriously, or understand how someone can make a living from it, but the fact is: we do. Those sites are my livelihood. They are my full-time job. They’re what pays our mortgage, and all of our other bills. They are three years’ worth of 12-hour work days, blood, sweat and tears.

And now I was being told that they were gone for good. My whole livelihood, wiped out in the time it took me to eat lunch. ( And, OK, to watch Neighbours.)

To answer the question I know everyone will ask: YES, WE HAVE BACKUPS. Obviously, given the importance of these sites to us, they are backed up. Even with back-ups though, it would’ve taken us a long time to restore the sites to the way they were.  During that time, we would be losing money: we started losing money as soon as the sites went down, and will continue to lose money until they’re fully restored. This is not a few pennies we’re talking about: it’s our livelihood.  As it is, we’re going to have to manually restore the last month’s worth of posts to all of the sites, and all of the advance posts I had been writing for my holiday in December are probably gone for good. Which means no holiday in December, clearly.

You can only imagine the day Terry and I have had.

With Heart Internet absolutely insistent that there was nothing they could do to restore the sites, I did the only thing I could think of in my panicked state: I started venting on Twitter. Naturally, Heart Internet didn’t like that much. In one of the many conversations we had with them this afternoon, they asked Terry to tell me to stop tweeting about them, “to allow them to concentrate on fixing the problem.” Obviously this begs the question: what had they been concentrating on BEFORE that? Naively, we’d assumed they were concentrating on restoring the data from our ruined server. The only inference I can really take from “stop tweeting so we can concentrate”, though, is that they were actually concentrating on… Twitter.

I did not stop tweeting. I WILL not stop tweeting, or blogging, or speaking freely about the things that happen to me, because that is my right. I understand that it wasn’t pleasant for Heart Internet to read the hundreds of re-tweets and comments about them. But the fact is, it’s not my job to protect Heart Internet from criticism by keeping quiet about something that was going to have such a huge impact on my business and life.  I related what was happening, as it happened, and I got a huge amount of support and advice from the people who follow me (and many who I’d never even heard of until today) – and that was a huge help to me, so thank you to those people.

Obviously, given that the sites are now (mostly) back up and running, albeit with posts missing, it turned out that Heart Internet WERE able to restore the data after all. Why did they call us at 2pm today, then, and say they were all gone forever, and there was absolutely nothing they could do about it? I’d love to know.

I will be writing more about this (um, probably much, much more) once the dust has settled and I know exactly what has happened. For now, our priority is to get the sites back to normal, so it looks like we have a long, hard slog ahead of us.

And I never did find out what happened at the end of Neighbours

 

[Edited to Add:  as the older posts are restored to the site, they initially won't have comments and the images will show as red X's. Don't worry, we are aware of this, it's just going to take a while to fix, unfortunately]

27 Comments

  • Beth says:

    I’m glad to hear you’re back up and running but have seen your tweets and these people clearly have no idea how to treat their clients. I’m kind of shopping around for a hosting company for mysf at the mo. No prizes for guessing which bunch of morons have made it to the top of my “Not if they were the last company on earth” list!

    I hope you guys get up straight soon.

    Bx

  • Rowena says:

    Bless you. I remember how awkward A Small Orange were when I left them and went to another hosts – I lost a lot of images and odd posts here and there but to lose EVERYTHING is devastating, and the thing is, its out of your hands for a lot of it.

    Hope you get back up and running soon! *goes to do a back up*

  • Ben Rose says:

    Glad things are getting back to normal and things could at least be partially restored.

    Moving forward, you need to rework your Disaster Recovery strategy and I’d recommend disk image based “bare metal” backups be taken, if possible. This allows you to backup not just the data but all the hosting server’s configuration etc.

    To put this into perspective, last time I had a “catastrophic event” I was back up and running within one hour…including the trip to PC World to buy a new HDD. As I also employ Transactional Logging, I lost no data at all – not even the stats from the page impression I got 1 second before the disk failed!

    In short, there are some good solutions out there. Feel free to ping me on the included email if I can offer any advice.
    .-= Ben Rose´s last blog ..Ecotronic Penguin Torch – No Batteries! =-.

  • Kristabella says:

    UGH! I’m so sorry to hear all of this! I’m glad you got most of it back. I’m REALLY hoping you find the advance posts because you need a vacation more than anything now!

    I think I need to back up my site tonight.

    Please pour yourself a big glass of wine! Hang in there!
    .-= Kristabella´s last blog ..DINAO Week 15 – The Halloween Edition =-.

  • siany says:

    Can’t believe them asking you to stop tweeting – what did they expect to happen?! That you’d just go “oh well never mind, I’ll start up again!” Some people will never understand that bloggers make money, but Heart Internet *Should*. They should be thoroughly ashamed of how they treated you and the entire situation.

    As for Neighbours… Donna’s “maybe brother” appeared. That was it. Granted it would have been more fun to watch that than have the nightmare you did, but you didn’t miss much :-)
    .-= siany´s last blog ..Lazy Girls’ Guide to Halloween =-.

    • Amber says:

      I know, it was appalling. When they told us what had happened they were just so casual about it – it took hours (and, I think, Twitter) to get them to even partially understand that this was not just some rarely-updated little blog being deleted, it was a whole business. I really got the impression that they expected us to just say, “Oh well, no worries, we’ll just start again from scratch, then!” And that whole casual, couldn’t-care-less attitude just made it 100x worse.

      Thanks for the Neighbours update too – Terry thought he was Donna’s dad, but he was obviously way too young!

  • Tracey says:

    I haven't heard of Heart Internet until now, but just for you I hate them. They are poo-poo heads.
    .-= Tracey´s last blog ..The Parsley Pot has moved =-.

  • Am absolutely livid on your behalf. Make sure if you see a drop in blog income this month, that you forward the bill onto Heart Internet. Utterly disgusting!
    .-= Erin Le Clerc´s last blog ..Pretty Vintage loves Snow White make-up tutorials! =-.

    • Amber says:

      Oh, I forgot to mention – they offered us three month's compensation! Which wasn't AT ALL insulting given the fact that they were telling me they'd just lost three year's worth of work!

  • Veta says:

    I can't believe the web hosting company treated you that way! It's pretty apalling, especially since even though they recovered most of the data and you have all the backups, it's affecting your winter holiday… maybe you should start inviting people to do guest posts for FP and your other blogs so you can have some posts stockpiled while you recover everything else?
    .-= Veta´s last blog ..music video features me [wearing vintage!] =-.

  • Mhairi says:

    Hey Amber

    I don't have any knowledge about any of this stuff but I can understand your frustration at the situation. I can't believe they ask you to stop tweeting so they can concentrate – was anyone from there even replying to you? Surely if they lose so much data twitter should be that last place they are looking too. I haven't been on twitter for a few days because of my work schedule so surely they can manage to tear themselves away from it to try and restore what they broke!

    Keep tweeting and keep blogging and I hope everything gets resolved sooner rather than later.
    .-= Mhairi´s last blog ..Weight Loss Update – the first in a while =-.

    • Amber says:

      They didn’t respond to me at all on Twitter, although I later found out they do have a Twitter account (@heartmedia). I obviously understand why they weren’t happy with the situation on Twitter, but I was surprised that they were even reading it when they’d just lost tens of thousands of pounds worth of data.

  • Nina says:

    Wow, this is B-A-D. I noticed that the site was down and that my reader got all confused declaring “old” posts as new ones. Thank heavens for backups. As the poster says: (Try to) Keep calm and carry on (and have some tea!) Best wishes!!
    .-= Nina´s last blog ..Things I have learned today =-.

  • Abi says:

    Phew thank goodness you got them back (although this really does beg the question of how they managed to do it when the sites were supposedly lost. Are other, less brave tweeters, perhaps just being told it's game over?)

    This all got me wondering how much responsibility a web hosting company actually has for your content/earnings. Really, they ought to be liable for anything you lost during this time, I think, but how much could they be held to account for if they really did NEVER return the sites? It's s a difficult question!

    • Amber says:

      I did wonder what would've happened to less persistent people – when they called to let us know what had happened, the matter was already closed as far as they were concerned: the data was gone forever and there was absolutely nothing they could do about it. They were absolutely adamant about that. As it turns out, that wasn't actually true (Did they even try to recover the data, then?) but if we had just accepted it, we'd be in a much worse position than we are now!

  • Rock Hyrax says:

    Wow – can't believe you got it together to write a piece after that day you had.

    Call me naive, but I'd have thought it was the web hosting company's responsibility to look after your data as well as the server hardware. Clearly they believe they might just be a tiny bit responsible for your data as it appears they do monthly backups. What's the point of paying for a business web hosting service if it's your own responsibility to keep a copy of your entire data? I mean what are you supposed to do, have a script to download all the comments every day? (I can't imagine transferring all those GB on a regular basis.)

    And the lackadaisical attitidue they had makes me suspect they thought you'd go away quietly and expect them to do their jobs properly – heaven forfend that they shouldn't be able to b*gger off home early on a Friday.

    In my opinion those people behaved amazingly unprofessionally and deserve any bit of damage that may have been caused to their reputation. Trouble is, I have a feeling this attitude towards backups is more common than we think – this isn't the first time I've been surprised by poor backup practices…

    And to think I've been considering setting up a blog and thinking, "I wonder who Amber uses, they can't be bad if she has several sites with them?"

    • Amber says:

      The only reason they had backups of Forever Amber, Fashion Police and Dollface was because they had transferred those sites to the new server for us when we got it, and they had copies of the sites from then – we had to remind them they had these, though. They had no backup whatsoever of Shoeperwoman – luckily we DID, and were able to restore it ourselves, or it would've been totally gone.

  • Virginia says:

    That's terrible news. Glad they were able to fix it in the end. It's really annoying when people say it's x, and then if you rant enough they're able to fix it. They should just fix it in the first place. I wondered if they would have tried as hard if it wasn't for your twitter ranting.

  • Hoshi says:

    Good news- I can see your images!

  • Amy says:

    Wow, what a nightmare. Wonder what their response would have been if they had just deleted a obscure blog and not a full-fledged business? Customer service is a dead concept.

  • annabelt says:

    How's it been since then? I'm with Heart as well and mostly they've been great but I lost all my sites for 6 1/2 hours last night – had some nail biting moments watching them come back bit by bit. There was nothing on the web hosting status page about it, which didn't help.
    .-= annabelt´s last blog ..Bizarre screenshot of the day: RSS directory? =-.

    • Amber says:

      Ours were all down for about 11 hours last night – or at least all of the WordPress sites were down, anyway. Really gave us a shock when we woke up this morning and realised everything was down yet again!

  • Anon says:

    Yet your still with Heart Internet, interesting

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>