Hi Guys,
In the other thread we're all pretty much having a huge whinge. Enjoyable eh? :D
However, it would be great to have a thread about what we learnt and what we can do ourselves to improve our systems.
Remember, this isn't a thread for complaints about Distribute.IT, you can ramble about that over in the other thread (/forum-replies.cfm?t=1711718). This is purely for sharing what you learnt, and how you yourself are improving.
Not all of us are huge hosts, some of us (myself) end users. So all our solutions will vary from provider to end user, big and small.
Please share what you learn't, and what you are doing to improve your situation/services for next time.
--------
I'll start... It never occurred to me that if DIT went down, or disappeared the difficulty I would have in retrieving my .com / .net TLD domain passwords. For .com.au we have AusRegistry which makes it very easy even if your provider goes belly up to recover. However for .com it's not as easy.
The solution i'm putting in place is a secure file, that stores all my TLD domain passwords and ensuring that the email addresses on the domains are working and current. In the event such an outage happens again, or my provider disappears off the face of the earth I have what is required to quickly move my domains.
Regards,
.Joel
I will add some cheap and simple solutions the list. This will work for any typical end-user website, and would only cost around another $10-ish month.
Get a second hosting account with a provider in a different DC. Pick one with the same control panel as the main. It doesn't need to be as big as the main account – if it becomes necessary you can usually get an upgrade quite easilly. On the backup site, set up your same email addresses, and add that as a secondary MX. Also use the backup host's DNS servers – so you are running with 4 nameservers. This can usually be managed simply enough via the control panels, but if you explain to the host how the secondary is working they'll be able to assist you.
Choose a reasonable backup schedule for the main site, it might be daily/weekly or even monthly if it's a static site that doesn't change too much. Also if you use IMAP for email, keep local folders and copy your mail to your hard drive at regular intervals (end of each day for instance). For most sites, you could easilly store the backup and any notes with passwords, domain info, host contact info, etc on a thumbdrive.
So, in the case of a disaster along the lines of DIT, you are well ahead of the game – no interuption to your emails (they will go straight to the backup host accounts), you still have 2 working nameservers you can manage, and you also have the option to restore your site on the secondary host and point your A records there temporarilly.
So, not expensive, not a lot of work, but a reasonable amount of redundancy.
I think important points are:
- Avoid single points of failure
- Assume you are the last line of defense
Something also important to note apart from redundancy/backups etc is also security. There are people/bots out there trying to get into your servers, and it's not a matter of if, but when. Doing simple things such as using non standard ssh ports, disabling root access, and changing passwords frequently dramatically decreases the risks of vulnerability.
Another cheap and simple solution is to take a regular full backup of your account and just download it to your PC. cPanel, as an example, puts all of your account configuration into that backup which makes it very easy for a host to restore it. Then, in the event of a disaster, you can easily restore to another host.
I think the point about the domain control is useful – either go with someone really big and well financed, or separate the two, or make sure your domain emails point somewhere external. (Judging the "well financed" bit can be hard if you are new in the market or new in business, I know).
The solution i'm putting in place is a secure file
I know you said "secure file" but it bears emphasizing – make sure you don't store password info in an unsecured file of any sort. Keepass is an excellent start, and lastpass or robopass take things a lot further with auto timeouts and full browser integration and almost automated password saving. (You'd still have to save the domain EPP keys manually with lastpass etc, but you could then be sure they were (a) secure and (b) always available easily.)
article about lastpass: http://blog.whitedoggreenfrog
<disclaimer>I work for a web hosting and application hosting company</disclaimer>
I think there quite a number of lessons that can be learnt at all levels. There's bits that DIT need to learn, bits that resellers / designers need to learn and bits that end users need to learn.
Tips for end users:
- Keep a copy of your domain registry key / EPP code.
- Take regular backups of your site and more importantly, validate them to ensure they're correct. Software like cPanel and Plesk have tools that can do the backups for you easily, then just FTP them to your office.
- Do some proper risk analysis. By this I mean drawing up a list of possible problems and what you'd do. Does your site run on code that has security updates? Does your site use custom code and is it protected from SQL injection? What happens if your host loses everything? What happens if you host goes out of business / down? You should have a plan for each of these and more (I can come up with about 20 just for hosting, yet alone other business risks).
Tips for resellers:
- Do all of the above, with more emphasis on backup validations and the risk analysis components.
- Replicate your backups to a hosting provider and not your office. Once you try to upload 50GB+ over a standard DSL connection in the event of a failure it'll lead to long downtimes.
- Actively talk to your hosting provider to find out what redundancy they have in their system and determine areas of weakness.
- Look at either having your sites hosted with more than one provider or at least have a standby system ready. Time how long it takes to restore all your backups, if you have quite a large amount of data it will take much longer than you think.
Tips for hosting providers (which I have taken on):
- Security is only as strong as your weakest point. There's no point having a good firewall if you have an easy way for hackers to get in.
- Better factor in bare metal restore times and how many servers are there as "warm spares".
- Look more into the security lockdowns between servers. While it's relatively easy to prevent access to servers behind a firewall, all bets are off once they find a way through.
- Look at specific application firewalls for all core services such as hosting management, billing etc.
- Better educate resellers and end users on what they should do to keep backups.
- Update DR and security plans to encompass all of the tips above.
The trick is going to be converting "lessons identified" into lessons learnt. I've seen on a number of occasions where people have lost all information due to lack of backups and then 12+ months later the same thing happens.
Hopefully people won't fall for the "it won't happen to me" or "it won't happen twice" sense of false security, when it comes to humans and computers they both fail when you need them most :)
Nice thread Joel.
I have a few servers with some small static and some CMS sites on, my clients have no idea on hosting and what is involved.
I thought i had a pretty good recovery system in place... (not)
I have been affected by the recent and more recent/recent outages at DIT and managed to get 80% of my sites back online using remote back ups to a dev server i own in good old USA.
The other 20% had DNS hosting with DIT so that caused an issue. The points you have made are educational and i thank you for them.
I will be changing my recovery procedure within the next 10 days (or as soon as possible) so this does not happen again.
Ill need to learn more about DNS and how to have DNS redundancy. everything else can be restored from back ups very easily.
Cheers
Mick
Doing simple things such as using non standard ssh ports, disabling root access, and changing passwords frequently dramatically decreases the risks of vulnerability.
All good advice.
Remember the DIT failure was not one of reliability but of security.
I am very wary of providers (at all levels) that are implementing strictly default mono-culture solutions out of the box without any variation or diversity. There is more than one control panel out there and there is more than one brand of firewall/router/switch/generic network device out there.
As an example 40% of the most reliable hosting providers are running FreeBSD, they do so for a reason.
Yet, how often do you recall seeing discussion of FreeBSD on the Whirlpool discussion forums? None, that I can recall recently.
Even better I'd love to see some mention of a REALLY secure operating system, OpenBSD, and some demand for hosting on that platform.
Go ahead Fool the script kiddies – do something different!
As an example 40% of the most reliable hosting providers are running FreeBSD, they do so for a reason.
And 50% in the top 10 running linux, but you didn't mention that.
and 55% in the top 20 too, with FreeBSD only 25%
I am very wary of providers (at all levels) that are implementing strictly default
On this subject, Cpanel and some other control panels, only run on particular versions of Linux, CentOS and one or two others. Which is why such a large number of providers use particular OS's but that does not stop you from using non standard SSH ports and the like.
Clients demand things like Cpanel and they don't like paying extra for backups, I have whitnessed it myself, where I have mentioned that I was setting up backups and the like and the general responce is Oh but you don't need to worry about that.
Those same clients would complain if their sites were down for a few hours.
where I have mentioned that I was setting up backups and the like and the general responce is Oh but you don't need to worry about that.
Maybe when there's a problem you should reply with "but you told me not to make them"... and then tell them how this would be a problem if it were actually true! Sometimes people just need a shock to the system to wake them up.
Maybe when there's a problem you should reply with "but you told me not to make
Ahh yes, but the backups are for my own data recovery as well, so I just ignored him, the client in this case is quite old and does not have a great understanding of IT systems.
Put it this way I would be relying on my providers too much not having backup systems in place, which a few people seam to be doing in this thread.
Ahh yes, but the backups are for my own data recovery as well
I meant pretend you didn't... scare tactics works well when making people realise how lost they'd be without them. And THEN inform them you actual have been making them.
If there's one thing I've learned from all of this, it's that automated backup jobs need to be run on the backup server logging into the source server and not vice versa.
n...
If there's one thing I've learned from all of this, it's that automated backup jobs need to be run on the backup server logging into the source server and not vice versa.
This needs to be RE-ITERATED !! too many people do not understand this and need to take note !!
If there's one thing I've learned from all of this, it's that automated backup jobs need to be run on the backup server logging into the source server and not vice versa.
Doing it this way, your "backup server" should only log into the "source server" using a backup account with only read access... Otherwise if your backup server is compromised then ALL your "servers" are compromised!
Doing it with the "source server" connecting to the "backup server" you need to make sure that there aren't any delete permissions from the "source server". Or compromised server = compromised backups.
Keep security in mind at all times, no matter what the setup and know the advantages/weaknesses and mitigate risks.
Get a second hosting account with a provider in a different DC.
I have done something very similer to this, I have a few redundencies in place within the DC, as well as offsite in a new DC. Additional to this I actually have a Mac Mini Running under my office desk whcih I have installed Virtual Box and its running CentOS with Cpanel/WHM.
I have set my main server to back up to my offsite and my offsite DC on my Mini.
Now, Idevelop alot of sites and stuff. I have an environment in which is replicated in 3 Physicall locatioins in addition to that my 3rd location (Mac Mini in the office) I have also have mirrored the HDDs as well as Apple Time Machine talking incremental hourly snapshot of the local system.
There are positives and negatives to this setup as per every system, I dont think there ever is a GARANTEED 100% redundency. But lets not be lazy and do what we know and can to make sure that the data is safe and looked after.
Great Post thanks for all the ideas...
Every Crisis is an opportunity...
:)
Further to all these great posts, the BEST backup solution is absolutely USELESS !!
I repeat **USELESS**
Unless you pro-actively practice restoring from backups at different points in time, so many times i've seen people with *multiple* backup solutions only to find there was a corrupt point at one that made its way through all the other backups...
only to rear its head when the "unthinkable" event happened and restoration was attempted only to find corrupt data.
only to rear its head when the "unthinkable" event happened and restoration was attempted only to find corrupt data.
I've heard this quite a few times too, although I've been lucky enough not to have this myself. I've been lucky enough to have all my server restores work (apart from a desktop PC, but that's a different issue).
- Disclaimer, we are a hosting/ASP/managed services business
We have not been affected apart from loss of domain registration/management... we use our own NS's and servers so have had no downtime. Thankfully.
What we have taken away is a small adjustment in the offsite data backup structure/procedures. Nothing particularly major, just some changes in directory structure and account password structure.
We have always had multiple layers of backups – on server archives at DC (full/incremental), off DC full backups (to another server in a totally different city), and physical offsite backups which are cycled around and are completely offline once taken.
I hope DIT come out of this OK... having to move domain mgmt to another provider will be a big job!
Id like to know what is a good backup proceedure for a Webdesign company.
Our hosting provider obvioulsy has backup in place but it means nothing if everything of theirs gets wiped.
Im wondering if i should be running something on each hosting account that automatically downloads a copy of the database and files in gzip format to our local server maybe running something like bulletproof ftp on a monthly basis or something ? at least its automated and some form of redundancy
Any other advice would be welcome for me and other small design companies.
Thanks.
D.IT talks about the attackers "destroying drive header files". Anyone know more precisely what they are talking about? What file system is involved here?
I think the number 1 lesson to be learned is ... if someone else hosts data of any kind for you, you should still take responsibility for backing it up. It's your data.
Ask yourself: Where is the master copy of this data? Where are the backup copies?
As far as D.IT itself goes, it is evident that customers will be more sympathetic if you keep them well informed. This applies as much to e.g. Bigpond as it does to D.IT or anyone else.
As such, a lesson for D.IT might be ... your means of communicating with your customers should be independent of the service that you provide to your customers.
But there are obviously a lot of more pressing lessons that will be occupying the minds of D.IT just at the moment e.g. how to keep backups safe.
great way to backup mysql db (will only work if the tables aren't too big).
- setup a new gmail account with a decent password
- setup a script on your server to export all mysql tables, zip them and email them to the gmail account via cron once every ~6 hours (lots of open source scripts around, will be easy to find one that suits).
as7r0...
Id like to know what is a good backup proceedure for a Webdesign company.
a company that strictly revolves around design only, i can reference ones that I know that are like this (i'm a dev / design / hosting house so we're diff) however, for the design business a friend runs.
They have a server in the office that all machines backup too 2 or 3 times a day (i forget which)
All work is set to autosave every 2mins, everyday at 4:30pm, the main server runs a daily backup, this backup is then sent across 2 external drives. both of these drives go with each director.. and are stored offsite at 2 different locations.
Shopping on price and complaining when the data you care about wasn't backed up.
Comparing hosting companies based on price!! Its like buying a Great Wall car and wondering why its ANCAP safety rating is so crap. Oh right, they saved some money on making the car safe.
D.IT talks about the attackers "destroying drive header files". Anyone know more precisely what they are talking about? What file system is involved here?
Not sure what they could mean this – it could be low level stuff – but they may mean the super block, but this recoverable even the deletion of all the partition records the disk is still recoverable. Basically if can raw read the disk you should be able to get something back. So I am interested in what was done and what file system they where using that seems to have made them give up. It a pet interest of mine Data Recovery, Oh and the girl is CSI I wish :). I have spent many hours digging though disk structures looking that vital just deleted file.
I run a few hobby sites and look after a couple of commercial sites.
Very early in the game I had a site disappear from the web due to hosting provider just stopping, I was all backed and moved the site in minutes once I realised the provider wasn't coming back. A quick change of the DNS upload a copy of the website to new provider and away we went.
Now I have always kept the registry and services provider separate as this prevents a single point of failure.
I also agree with the comments that at some stage you will get hacked its only a matter of when and then how good was your disaster recovery plan.
The ones that scare me the most are zero day attacks and they get in and you have no idea how they did it and what to patch to stop the attack.
The commercial site I look after is planning a major update and many of the points raised here have given me much to think about. I had been pushing for a remote fall over solution as mention earlier in this thread and the info here has given more points to add the case for this solution so thanks for the great ideas.
I also agree with the comments that at some stage you will get hacked its only a matter of when and then how good was your disaster recovery plan.
Sure, not every site will be hacked. The thing to be aware of though is you COULD be hacked at any point (through your web host, the server, another client on the host, etc). The point is it could happen to anyone at anytime and without notice.
Whilst there are a plethora of security options available to companies and users, it's important to note that the shared web hosting is one of the most insecure platforms – a network security is only as good as it's weakest link, and unfortunately a lot of people are running very insecure software on their sites, opening the whole host to potential attack. Web hosting by nature is prone to risk.
Anyone know more precisely what they are talking about?
Sounds like the hacker probably DD'ed 1's over the top of the disk starting from the start.
http://en.wikipedia.org/wiki/Dd_%28Unix%29
Contrary to the urban myth that the CIA could retrieve data that has been overwritten multiple times, I think overwriting it once makes it infinitely impossible to retrieve :(
On Wikipedia it says there is a joke that dd stands for disk destroyer.. or death and destruction...
Do you guys think the number of customers they had on each server should be something to avoid? They lost 4 production servers which equates to 4800 web sites.
What are the implications of hosting this many clients on one server? I couldn't even begin to contemplate how I would handle 1200 clients emailing me when their server goes down!
Another issue that DIT came across was both their production servers and backup servers being hacked. Could this be prevented by simply unmounting the hard drives after a backup has been completed? That way the data is only at risk during the backup and is offline at other times.
Sure its not as secure as an offsite backups but something thats simple to implement and can make restoring a little quicker too.
On Wikipedia it says there is a joke that dd stands for disk destroyer.. or death and destruction...
especially if you've set it to use multiple runs with /dev/urandom as the data source!
If you're lucky you might get something back to if you spend $1,000 (minimum) for DR expert recovery per disk (and we're talking raid arrays here, not single disks), but then you're not going to guarantee anything.
Could this be prevented by simply unmounting the hard drives after a backup has been completed?
If the drive was mounted using a script on the server that ran at a specific time, the hacker would have access to the script too...
Sounds like the hacker probably DD'ed 1's over the top of the disk starting from the start.
http://en.wikipedia.org/wiki/Dd_%28Unix%29
On Wikipedia it says there is a joke that dd stands for disk destroyer.. or death and destruction...
Yes that would do it in spades simple clean and in the end very destructive.
The only thing that would save you in this case if got to it before it overwrote everything, its is quite slow operation. Even then it would still almost be an impossible mess to recover much of value from a disk left in that state.
I do believe the CIA start looking at electron microscopes at this stage, and actually inspect the way the disk has been modified by the magnetic read/write operations. This is where the multiple read/write DDs are needed. While it may be hard to read a disk with software once its been DD wiped, there are other hardware/mechanical ways to access the data.
Although, I would dare say this is beyond the scope of D.IT and well beyond their budget. Why they never had offsite/offline backups is a mystery to me.
Also, in regard to the mount/unmount scripts. If they were activated via the backup server and not the production servers, then hackers might not be able to get to them.
For me, it would have been a shell script that opened the network device, connected too and copied contents from the prod server, then closed the network device. Do this on a cron job, staggered across backup servers. Once that operation is done, you write the data to external disk/tape and then take that out of the building. with N machines doing snapshots like this, you have almost continuous backups with small time frames between each machine.
Unless the attacker was physically at the machine, or happened to be in the system during a backup, the worst case scenario is one backup server, out of n being compromised.
If the machine is not on the network, it cant be touched remotely. Put the machine in a cage, with a lock and some fancy security cameras, and its pretty impossible for the hacker to gain physical access.
Hey dude, um what your recommending is for inhouse backup... im talking about if have all our websites hosted in the cloud.
So if i have multiple accounts in the cloud what is the best way to back them up.
Hey dude, um what your recommending is for inhouse backup... im talking about if have all our websites hosted in the cloud.
Gar. THIS IS NOT CLOUD COMPUTING...
The media has completely destroyed SaaS cloud computing with its dumb reporting.
Cloud services are simply a way of distributing software and data without having to install the software locally. If we were to draw comparisons to cloud computing, it would be more like mainframes and dumb terminals from the 80s.
Added benefits of cloud computing include scaled resources and SLA agreements. Do you really think Amazon or Apple would allow shit like this to happen?
We are talking about HOSTING here. Unless your company enjoys poverty, hosting is the only way to serve up your content reliably and cost effectively to other users.
The problem with D.IT is that their level of service in regards to data integrity was poor. My post above is a suggested way for scripting a backup server. It was simply highlighting that it can be done relatively safely. Unlike the assertion previously that suggested it couldnt.
To back up your hosting account, refer to many of the posts on first page that suggest having two hosting accounts, using cPanel, etc to do site dumps and even set up your SQL server to do backups on a regular basis.
Even with a dedicated server in a data centre, someone is still hosting that for you. Just you are more responsible for the hardware than using VPS or whatever. Something that D.IT was offering I presume.
Gar. THIS IS NOT CLOUD COMPUTING...
You could also blame Apple. The nuffs at one of the Apple shops in Richmond were crapping on about Cloud stuff the other day (to even bigger nuff customers). "You can get all your apps across all your devices in the cloud" was one particular quote... I guess I'll be loading up Photoshop on my iPhone come iOS5!
What this whole situation really represents though is that people are under prepared for situations like this, even the bigger providers. I still remember when iPrimus had that outage with regards to the backup generators, those were fun times as well.
I still remember when iPrimus had that outage with regards to the backup generators, those were fun times as well.
I remember Telstra's outage back in 2004 I think...
Generator's & UPS were getting maintenance on the same day... someone crashed into the main power feed to the building.
Bad month >.<
I always back up data to Amazon S3 with a program called s3cmd which is automatically run from a cron job.
It's work really well and I know my data is as close to 100% safe as possible.
Cloud services are simply a way of distributing software and data without having to install the software locally. If we were to draw comparisons to cloud computing, it would be more like mainframes and dumb terminals from the 80s.
Added benefits of cloud computing include scaled resources and SLA agreements. Do you really think Amazon or Apple would allow shit like this to happen?
it would not surprise me at all
http://viodi.com/2011/04/22/
'
For several years, we’ve been pounding the table about many of the potential problems that Cloud Service Providers have “swept under the rug” or ignored. These include failure/ disaster recovery, SLAs, security, lack of standards, vendor lock-in, etc. During the session “Everyone can now afford a Disaster Recovery Center*” at the 2011 Cloud Connect conference, the speaker stated that disaster recovery could be solved by transferring the work loads from effected cloud data centers to other data centers owned by the same Cloud Service Provider. He gave an example where cloud outages in San Francisco, CA resulted in the jobs transferred to cloud data centers in London, England with data being replicated there. I strongly challenged the speaker about the complexity of doing this. In particular, the resulting extra processing load on the servers in London, data replication issues and network bandwidth saturation. He danced around those problems and remained confident disaster recovery would actually be a strength, rather than a weekness of cloud computing. Wonder what he thinks now after the Amazon cloud outages?
'
lets stop with the cloud kool aid and hype
+1
Complex systems have complex failures.
wish more people understood this, the number of times I have people saying "nah its fine its in the cloud" is concerning to say the least!
"nah its fine its in the cloud" is concerning to say the least!
I'm not really sure of the arguments for and against the cloud, but I can say I feel very very secure having my data on Amazon S3.
The data I have backed up locally I feel less sure about. I have duplicate HDD and thumb drives with data but I can say I've never lost a single byte of data from the cloud but I have misplaced tiny thumb drives and thrown crashed HDD in the garbage.
what @gravylicious said and:
Content monitoring:
Email a report of new directories and files created daily.
As part of your site monitoring, use html monitoring to check for a hack
DNS
Use a 3rd party distributed dns for your domains and make sure their control panel is distributed also :-) This way you can point anysite anywhere without relying on changing nameserver records and if you change providers or registrars it doesn't effect your sites.
Operating Systems
No current OS is more secure than the next. It comes down to who is configuring it and running it to how hardend it is, then what's running on it.
It's WHAT is running on it and what access that software has that matters. Hackers rarely start at the OS level, they work to it from easier targets.
Version Control
Tell your customers to use a version control system, or sell it to them and host it offsite
Looking through the posts from when I joined this discussion the attack seems to have come in though an ssh session, which is why those with colo haven't been able to access with ssh for a while.
This begs the question does any one leave ssh open on the standard port or do you close it down completely.
On one set of servers I shut ssh down about twelve months ago as I got sick of seeing repeated passed scans in the logs.
I know in some cases ssh is required but is it worth the risk?
I hope we get a full technical discussion of what happened so we can all inspect our systems and plug the holes.
I know in some cases ssh is required but is it worth the risk?
It should be at least be firewalled to the office's IP if it was determined it needed to be left open.
This begs the question does any one leave ssh open on the standard port or do you close it down completely.
If you're providing servers, clients expect SSH access, so it's kind of difficult to just close the port for the whole network ( just witness the complaints that were happening about it)
the individual admins should have locked SSH down on their own vps/vds/dedis though. like a different port, answering to only a single IP, access control for specific IPs only, firewall rules, etc.
Change SSH from the standard port 22 to some other spare, higher, port number. Use that new port as your own standard across your network. That way you can block port 22 at the hardware firewall and there will be no more filling of log files with failed attempts to ssh on port 22.
Oh...and if possible only allow access to ssh from a very limited IP range (or better yet allow on a single/dedicated IP).
Cheers.
Change SSH from the standard port 22 to some other spare, higher, port number. Use that new port as your own standard across your network. That way you can block port 22 at the hardware firewall and there will be no more filling of log files with failed attempts to ssh on port 22.
Oh...and if possible only allow access to ssh from a very limited IP range (or better yet allow on a single/dedicated IP).
Cheers.
Thanks for this I have done this in the past but I had 22 open for a short while to do some intra system testing and was rather surprised how soon the probes came shut the port down as I really didn't it open.
If isn't open it can't hacked.
I used to be of the opinion you must always have ssh access to your machine but I have found a lot of jobs can be done with out this level of access.
Recently a company I work for employed a web design team to up date and CMS our website. Now this was all done without the web design company even getting access to the cpanel. So work can be done with limited access.
Well I decided after my site host screwed me to become my own site host on my own paid for dedicated bandwidth.
Out of that I have had a few clients ask me to host their systems as we had used the same site host before. I just supply the bandwidth and they supply their own PC server and IT staff to set it up, and I supply their own dedicated fixed IP and UPS. 10mb up 100mb download lines and of course no capping.
So I have my own server for myself. Not shared with other clients. Backups made daily. Fixed IP and hopefuly every other IP blocked from entering the server. I am not an IT person but have an IT consultant. Even then I should probably not rely on him alone but since he setup my server in my own office a few years ago never had any problems.
I am sure there are good site hosts but for me never again. I run a small business but having my own server in my own office with proper backups done to more than one hard drive daily keeps the worries away. I can get backup online in a very quick time just by changing the hard drive if I had a problem.
PS I am not in Australia but have business interests there.
If you think you can't afford to host your own server.. think about not having one at all when it goes down and you have no way to get it back up online.
Im wondering if i should be running something on each hosting account that automatically downloads a copy of the database and files in gzip format to our local server
I am also a small reseller and would also like to ask the same thing that as7r0 wrote:
What's the best way to automate a backup of all your clients files from WHM or Cpanel?
I currently use automysqlbackup.sh to automatically backup the MySQL DB daily, weekly and Monthly.
But I would dearly love to set up an automated file backup too.
Going in to each account's cpanel weekly and performing it manually is just not practicle.
Please tell us what solutions you use :))
Thanks
What's the best way to automate a backup of all your clients files from WHM or Cpanel?
Currently there is no way to do a full "reseller backup", but I do believe cpanel is working on a method for a future release.
You could try this
http://www.totalchoicehosting.com/help/id207.htm
I've never tried it so can't comment on it, but it might fill the gap in the meantime.
For several years, we’ve been pounding the table about many of the potential problems that Cloud Service Providers have “swept under the rug” or ignored. These include failure/ disaster recovery, SLAs, security, lack of standards, vendor lock-in, etc.
Exactly. When sling media decided to migrate their servers to cloud servers last year in August they had no backup in place should a failure occur.
Occur it did, they screwed up royally and most people using sling media lost connection for up to a month. No recovery and no backup plan.
If there's one thing I've learned from all of this, it's that automated backup jobs need to be run on the backup server logging into the source server and not vice versa.
Reading this really made me think. I thought I had a solid backup strategy. Incremental backups to one Amazon S3 region, a mirror to another region and a rolling 7 day snapshot . . . but it the source server became compromised . . . everything would be compromised. So obvious, but still, I missed it. For those of you using Amazon for backups, I suggest you have a look at Amazon's "Identity and Access Management" rather than using root X509 certificates or authentication credentials. http://aws.amazon.com/iam/
Currently there is no way to do a full "reseller backup", but I do believe cpanel is working on a method for a future release.
Sure you can do this easily in WHM. You can set this up to be completely automated to create daily, weekly or monthly backups of your clients home directories which includes all their web hosting data, logs emails etc
This can be stored in a separate hard drive or sent via ftp to another server.
Its also really easy to restore the data in the event of data loss as you just restore the customers entire home directory