That’s right 1 silly mistake crashed my business for 4 days! No I’m not talking about just one of my websites, I mean all of the ones I have on my 1and1 webspace.. Fortunately I have other online business on alterative servers so I didn’t lose everything, but the majority of my online products and blog were gone in the space of a few hours!
Let’s begin with a short story of how it all happened
Monday night I received an email from 1and1 hosting to alert me there had been a malicious file uploaded to one of my websites. This website was currently under construction and had been for a few months due to alternative products and ventures taking precedence. I was offline at the time, but I checked my sites such as this blog, the Bloggers Roadmap, Product Development Control, eBook Cycle, Dan Sumner where I have all my recommends affiliate links (All of them) and a couple of other blogs and squeeze pages. Everything was ok, so I thought it had been contained in the domain that wasn’t ready. No big deal I told myself! (Epic fail!)
I woke early in the morning on Tuesday and logged on as I was a little concerned to what end of damage these files had done to my Video Training product. What did I find? Well what came next was very bad indeed, heart stopping bad! I found a blank white page with ‘Access Denied’ in the left hand corner? I tried logging in to my admin panel – no good! Then it hit me.. What about my other sites?
As I went through the domains of the sites on my webspace, I found they all displayed the same text ‘Access Denied’, ‘Access Denied’, ‘Access Denied’, ‘Access Denied’, ‘Access Denied’! It was at this point panic set in.
Ok, what was I to do. When this happens you need to think logically and try to diagnose the problem, so I called 1and1 for support. At first they were very helpful and told me that I should change all my passwords for login, databases, email, FTP, site admin and any other online related passwords. It was at that point I asked what about my sites? 1and1 told me their security team would be in touch shortly. So I was stuck for a few hours while they looked into it. 12 hours actually!
Late Wednesday night, 1and1 send me a file with all the corrupted links, over 600 corrupted files. Whatever had hacked me had spread like a virus across my webspace and disabled all my sites completely. Just my luck as I had just released my first podcast (check it out if you missed it).
The next day I called the 1and1 security team for a solution to my problem. This is where it all went wrong!
I asked the customer service guy what to do, he didn’t know? I asked him for a backup of all my sites, he said they had been corrupted in the backup and that was no good. Then he suggested I delete all my webspace and start again! For some reason he put the phone down? I must of offended him..
I tried again, only to hit another roadblock, which resulted in this particular member of the security team asking me if I had a web developer with me. To which I responded I never, then she said to me “well I think you should get one!” For some reason she put the phone down? I must of offended her..
I was tearing my hair out literally, so I had one last ditch attempt at 1and1 tech support who said a backup would be with me in 24 hours time. Completely disheartened and very low, I put the phone down without any real help to why I had been hacked, how to fix it or where to turn. 1and1 did actually try to sell me a security package, but said they couldn’t fix the current issue. So once again I was left where I started in my own business, with nothing and no help! Time to get busy!!
With everything broken, I took at look at my webspace and the accumulation of crap I had on it. I looked at what was making money, what I used and what I could use. I created a criteria like this:
- Profitable
- Future profits
- Trash
I set to work removing everything I had, creating a cleaning house! Domains were cancelled, emails removed, blogs removed, members area’s removed, AdSense sites removed, old products removed. I basically cleaned up my webspace, and was left with my core, profitable business. Everything that wasn’t making money or building my mailing list was now gone or redirected.
I had already spoke to Dan Thompson of D9 hosting about a move when I was up and running again. This is now in the pipeline and should be transferring shortly. This is all due to the complete lack of support from 1and1. After 8 years of hosting, this is how much customer loyalty means? No thanks I’ll move on, I obviously mean very little.
My next step was my good friend and tech wiz Dave Nicholson. Dave is a genius when it comes to web code. I know a little about coding and building blogs and websites, but Dave is the Jedi Master. I sent some files and within about 5 seconds diagnosed my problems. I had 600 files to fix! So I set to work and 16 or so hours later! My site sites are back again! I am behind now on daily tasks, so if you did email me sorry I will get back to you.
How to avoid this type of disaster and what you can learn from it
The rules are simple. With the invention of push button websites, such Optimize Press and WordPress security flaws are more common that ever. So you have to be vigilant and stay on top of all your sites.
1. Update the sites you run, the ones you don’t use or need any more delete them from your webspace. Back them up by all means, but don’t leave them to hacker back door attacks
2. Update all plugins and wordpress versions
3. Use a security plugin such as Better WordPress Security
4. Back up your site regularly or use a backup plugin if you use WordPress, use a backup plugin. I would still download your server once per week to be sure
5. Get a good hosting service who WILL support you with your sites and place them on separate control panels to minimize damage
6. Try a third party security team such as Sucuri to help you
The bottom line is, don’t leave yourself open to corruption. As much as you may think it’s not going to happen to you. It can and it will if you are not super careful. Times are changing when it comes to websites and done for you technology and that means security issues. When you are relying on websites, themes and plugins created by different development sources, updates are not seamless which can lead to security holes. Be careful.
As well as the above, I do recommend evaluating your business once per month. The simple tasks such as:
- Cleaning up your webspace
- Managing your mailing lists
- Removing what isn’t working to free up time for what is
- Trying new ventures to improve your business
- Moving forward
Please take note of the last three items from that list. You can’t do these if your webspace has been hacked. Take care of your websites and webspace the right way and it will run itself.
I hope this post helped and I hope my experiences help you in the future.
Subject for comments – what are you doing to protect and manage your business. Do you evaluate your businesses?
That’s great advice thanks Dan,
I already use a plugin called backup creator but I’ll definitely take a look at your other suggestions. I’ve lost important work before as well (not as much as you by the sound of it) and it really does hurt.
Need to get on top of this!
Cheers
Jon
Thanks Jon. There are a lot of plugins out there that do pretty much the same job. I won’t be letting it slide from now. Thanks.
Hi Dan,Great post. When I saw the title and started reading it, I thought \’this guy is telling me a sad story to scare me into buying some super security product\’, but I was wrong…I got to the end and there was nothing there, other than very good sound advice. I\’m sorry if I doubted you for a second. I will certainly follow up on your recommendations.Cheers,Richard
Thanks Richard. I never thought if it as a sales pitch 🙂
Criky I’m so pleased you got sorted Dan
1&1 suck big time, I’ve heard so many bad things about them I’m surprised they are still up and running!
Kim
Thanks Kim. I have been with them for a long time and had no problems. Usually dude to me fixing most things myself. But now it’s time for a change. Managed server I think?
Thanks Kim
Dan
I had one web site that was attacked and all it showed was a black flag like those used in Muslim jihads. I had no idea what to do, but I had a friend who worked with my then hosting company and they got it sorted.
The site wasn’t making me anything, but it was worrying to see how easy it had been to get hacked and how difficult it was to resolve it!
The one thing I was told was to change my WP access name from Admin to something more difficult even my name in both upper and lower case!
Glad you have got it sorted out now and are back in business!
DaveT
Yeah a password change is the minimum now. My friend had the same hack as you. These are not so hard to get rid of but are a pain in the ass.
Look at security plugins and backup plugins to. They do help.
Hi Dan
Sounds like a real headache you have had over the last few days. I think the move over to D9 will be a wise one. I personally use D9 and their support is excellent. This post has just prompted me to do another server backup so thanks for that.
Can I just say thanks for making the time to write this post warning us all about the issues you have had because not only have you been put behind sorting this headache out but you have also given back something to others from you bad experience.
I am going to print this post of & keep it on my desk as a constant reminder of how important it is to make regular backups & clear out any unused trash from my server.
Regards
Rob
Yeah Rob, it was a real headache and a challenge to fix it all. I have reminders setup on my calandra not to back up my server once per week. Glad this helped Rob thanks for reading.
Hi Dan
Ever since Randy lost Wizard Responder to hackers I’ve avoided 1&1 like the plague. I’ve stuck to more helpful providers. Hostgator are not as good as everyone once claimed either. Another provider I’ve managed to avoid.
Thanks for the useful advice regarding what to do if the worst ever should happen.
About once a month I clean everything I can think of on my computer and back it up. Then check everything I can on my hosting server and clone that too. The reason for checking everything first is to try to avoid backing up a problem and having corrupted back ups. It’s a real pain but I suspect not as bad as trying to sort out the kind of problem you had.
Cheers
Tony
Hi Tony, yeah I think an evaluation of providers is now in order. I will probably go with D9 as I know Dan Thompson personally and he is awlays an email or sms away. He backs up daily to.
Hi Dan,
Glad to see you are up and running again. I agree with your thought on spring cleaning our webspace now and again and focusing on our websites which are successful. I know my own webspace must be cluttered up with long forgotten test pages and test sites which really should be deleted. Your problem has given me a wake-up call to have a good tidy-up.
I guess though that there must be many of your students like me who have quite a few blogs and find it time consuming to keep them all up-to-date. Is there any software you could recommend which we could use to automate multiple blog updates; or would the use of such software be a security risk in itself?
Also, regarding backups; if one of your students, such as myself, has their webspace hacked, but has been good and has recent backups downloaded daily to our computer, how can we be sure that the backups we have do not themselves contain some hackers code (or back door) which the hacker installed on our webspace several days before actually carrying out the defacement (i.e. how do we know our backups are clean)?
Although you are frustrated with 1&1 I must say that I am impressed that they did actually warn you that your webspace has been hacked, and could give you a list of the actual hacked files. This is way above the level of service some of my friends have received from other webhosts.
Glad you are back up and running!
Ian.
Hi Ian, I actually have a small bit of webspace with D9 for testing. I’ve had it for ages and it works for me to use that. I simply delete it now and again. WordPress now actually updates itself, however it’s the third part stuff you need to stay on top of such as themes and plugins. I don’t know of any software that does this.
As for the hacking to your backups, it’s s good idea to check out the problem and then check the backups for the same issue. I had the same problem with 1and1 They only have a 7 day instant backup which was corrupt! This was my main issue. It never blew up until Monday, but the corruption was in place before that. I had backups so I was able to restore some of my sites.
There is software out there such Sucuri.net which D9 recommend using and also 1and1 has something called Site Lock which is similar. They scan your servers for viruses. This maybe worth looking into. Especially if you have any affiliate links. I lost all of mine, for how long I don’t know but I seen some commissions come in today which I haven’t seen for a while, so maybe they have been missing for a while 🙁
In a lot of ways 1and1 were not helpful but equally helpful at providing what I needed to fix the issue. For a non tech person, I would say they would be a nightmare!
Thanks Ian. let me know if you have any other questions buddy.
Hi Dan Thanks for the advice, it must have been a nightmare
glad you have sorted it out.
Regards Gerrard
It was, and I need to secure the future. I have worked about 20 hours straight trying to get back on track and I’m still fixing stuff now such as old emails and logins. It also puts your business on hold. So you are building rather than fixing!
Hi Dan,
I have BWS on my other sites, but for some reason not on my P2S site, which is now rectified. You may already know that this plugin was acquired by iThemes recently and is now called iThemes Security. Again, you probably already know that the backups generated via this plugin are for databases only and to include all files, images, themes, plugins etc they recommend another plugin called BackupBuddy which costs upwards of $80 depending on how many sites you want to protect. Do you or any of your commentators think that this is necessary?
Cheers,
Richard
Hi Richard,
I would say it’s a lot of investment for me, for the simple reason I know how to fix most problems or have the support of people to do it. As long as you keep all your databases backed up and files downloaded, its fairly simple to put it all back together. Some folks don’t know where to start, so for these people it may be worth it. I backup my databases and have some security plugins to.
Personally I am a little worried about OP1 theme. I’m usually a thesis person, which I have had no problems with. I may revert back in the near future.
Hello Dan,
Thanks for this information. I will take your advice as far as of backup.
Perhaps this is something you might want to write more about in the future is that of Internet Security and File Backups. It seems to be the focus of the news even as I write this.
Is there a site that warn you of the most recent virus attack?
I saw one with the Heart on the news last week.
Your information is important and I’m glad you got it back!
Thanks Thelma, I will put a full security post on the list. For the mean time its always a good idea to keep an ear to the ground for updates and security updates from hosts, FB groups and forums.
Hi Dan,
What an awful mess and I know exactly how you feel. I have been super-lucky with my webhosts (Free Virtual Servers) I’ve been with them for 12 years now and their support is fantastic, I’ve never had to wait any longer than an hour for them to help me fix things on my sites and their support team have taught me so much over the last 12 years, without charging me any extra for the knowledge! I love them to bits and I know that I can rely totally on them to give me complete support at all times 24/7.
It’s a good thing my hosting is so good because I too had the same problem on three of my 12 websites and my hosting company alerted me immediately and changed the logons and passwords for me, removed the infected files, cleaned up my sites and sent me a full report on what they had done to eliminate the threat. The sites remained out of action (about 2 and half hours) until I went into each to ensure everything that was supposed to be there was still there and that there were no extra files which may have been missed by FVS.
As it happened the three sites in question were actually satellite sites for link juice purposes and none were actually making any money on their own so it was easy enough to get them back up and running – this time of course, with better security.
It’s a real stomach churner when you realise someone is breaking into your sites; but having a super efficient and helpful webhost means the difference between your sites being down for days, or even weeks as opposed to a couple of hours. I’d recommend FVS to anyone – and I frequently rant about how good they are – I’ve never come across such helpful and sympathetic hosters and programmers anywhere else in the world and I would never even contemplate moving from their hosting.
My normal policy is to download weekly backups to my pc, once I have run the security software provided on FVS cPanel – this only costs me an extra £6 per year and is the best value for money I’ve ever come across.
I’m glad you managed to salvage everything and get your sites back up and running but it sounds like your hosting company leave a lot to be desired compared to other more reputable companies.
Thanks for sharing, Dan. Hacking is the new robbery, assault and battery and it really pays to have the best support from your hosting company as well as the best security plugins you can find. When it comes to security plugins I personally use Bulletproof Security on all my sites now and have never had any hacking problems since my disaster last year – I should probably also mention that the three sites which were hacked were the only sites on which I had NOT installed BPS so I think that says a lot for the security plugin too since all my sites are held on the same hosting account.
Cheers Dan.
Thanks Caroline, that’s a mighty comment and worth reading! Thank you for the info you provided, I will take at FVS and see what they have to offer. You are right about hacking being the new robbery. Stealing my affiliate links without telling me..
Thanks
Ugh man, so glad you have everything back.
The bright side of this experience is that not only you learned the lesson (through a hard experience) but you were forced to clean up your webspace.
That’s something I have been neglecting myself for a long period of time but something that I must do as well.
So happy that you’re back on the game and I’m pretty sure this is a great lesson for all of us as well.
Cheers man!
Sergio.
Heyyyy Sergio! I did get everything back, but not without some pain. I can see why some newbies would throw in the towel from something like this. You are right though buddy, I did learn a very valuable lesson from it. I also know I can work for 20 hours without much of a break to lol.
Hi Dan,
I have my .co.uk domain names with 1&1, but had such a terrible experience the one time I tried to use their support I decided to steer clear of them for everything else, in particular hosting.
The hacking I have experienced was on sites that are all on a single account with addon domains.
I think they somehow got into one and accessed numerous others.
In some cases the index page was replaced with a black one and colourful “XYZ was here” type stuff – in the source code you could actually see the person supplying the pages advertising his services!
Those ones I was able to fix by re-uploading the index page.
I also had the content of a couple of niche blogs removed.
I had backups (like Jon I also use Backup Creator, with both regular automatic backups that also go to Dropbox as well as manual backups before and after e.g. theme updates, and have WP Twin too as a back up to that so to speak, because it runs via FTP instead of inside WordPress), but the first time I couldn’t get the databases to link up.
Fortunately the hosting company (Midphase, a recommendation from Jeff Johnson many years ago) were very helpful and we got it sorted.
One thing I did do then though was go through all my WordPress sites changing login details and adding all the security I could find.
And I keep the Backupcreator backups for quite a while in case I do end up with backups of corrupted sites and can then go back further.
As I have developed the habit of keeping Notepad copies of my blog posts on personalised sites (as opposed to VRE sites), if I did have to go back I could still add more recent posts back in.
I meanwhile also use sftp instead of plain ftp when going into the back end of my sites with Filezilla.
I also created my own mindmap checklist for setting up new blogs as securely as possible, based on what I researched from various sources.
Even then, I had hackers back in again later because I had missed something.
It wasn’t WordPress, but a couple of done-for-you sites I had put up years ago, and they were somehow spamming or injecting something into the comments that I had never thought to switch off.
The result was massive resource use at certain times that resulted in all the sites on the whole account being temporarily disabled by the hosting company until it died down again.
They were able to tell me where the resource use was coming from (it was in the log files, but I wouldn’t have known where to look myself), and I took steps to stop it happening again.
And fairly recently I just deleted a whole site and switched the DNS back to Namecheap rather than bothering with it.
The host had created a file listing all the corrupted files, but in this case I figured I wasn’t going to do anything with the site so I bit the bullet on that one.
(It was in German, named after a physical magazine I published years ago in Germany, but most of the content had been on old, meanwhile inaccessible floppies except for two articles about TV documentaries in New Zealand and Tahiti/French Polynesia I was involved in making, and digitising the rest turned out to be a utopian idea!)
All in all security is something you can never be complacent about.
Regading what Ian asked about something to upgrade WordPress and the plugins on all your sites at once, in the past I have used a program by Shannon Herod but after my latest attempt decided to stop using it as it didn’t work on most of my blogs.
I did get hold of WP Pipeline, which has a master/slave system for managing multiple WP sites at once, but I must confess I have never set it up.
I also had Blogmatrix, which JP (Schoeffel) acquired at some time, that you install on your server.
Unfortunately that was one of the ones that got messed up the first time I got hacked and I recently deleted it from the server as I hadn’t been able to fix it – think I need to go back to the drawing board with that one, see if they are still supporting it and potentially start again if it looks worth the time.
But I think all of the recent changes at WordPress, including the automatic upgrading, may have changed the ground rules for a lot of these applications.
What’s that old saying from one of the ancient Greeks? Nothing is as certain as change, or words to that effect!
In any case, I’m glad to hear you managed to get on top of it in the end, and your post is a reminder to us all to always remain vigilant about ALL of our sites.
Cheers,
Paul
(“Kiwi” in South West Scotland)
Thanks Paul, you seem to have had similar problems to the ones I have had. I will be changing hosting accounts next week for the simple reason, this is ‘MY BUSINESS’ and it should be at it’s most secure at all times.
WordPress auto update is a great addition and saves a lot of time, although you now have to manage the plugins and ensure they are all up to date when you receive the email from WordPress update. All in all it’s a pain in the ass having to babysit your sites, so we all need to give them the best and automate as much as possible.
Thanks Paul.
Hi again Dan,
Thanks for the helpful advice. Your idea of testing websites on a different webhost (D9) from your main sites (1&1) is something I have not thought of before. Thinking about this it does make sense from a security point of view as the temporary test sites are more likely to be hackable.
I notice that in your blog post (point 5 of “How to avoid this type of disaster”) you advised placing our sites on separate control panels. How do we do this? Are you suggesting we host each website on a different webhost (i.e. some on D9 and some on 1&1)? Or is there a way to have multiple cPanels within a single hosting account?
Ian.
Hey Dan
I’m sorry to hear about those pesky hackers getting into you site.
I always have a sinking feeling when I see this happening and wonder whether it’s going to be me next.
I do have security in place, CloudFlare, WordFence and a backup plugin. You just never know though whether it’s enough.
Glad to see you’re back up and running though and thanks for sharing what happened so we can all learn from it!
It was bad Tim. Really bad actually, but I got it sorted. I don’t think anything would have helped may main blog as it come through another domain via my webspace! Scary the damage they did.