Checklist for Cloud Service Operations

I knew one software company that failed their SaaS transition because they chose to cut a few corners with the operations. Since they were software engineers, they did not really want to spend time on such mundane tasks as security, auditing, backups and so on. One day they let a disgruntled employee go, the guy went to an internet cafe, logged into the hosting account with the shared admin credentials, and deleted all customer data.

There were no backups or data replicas to bring the data back, no personal admin accounts or procedures to prevent such an incident from happening, and even no monitoring to learn about the issue before customers did. This was the end of this SaaS application – it just never recovered.

Agile DevOps in the Cloud - Session recording from WSO2Con Asia 2014Cloud business is more than just putting some code online (and collecting money ;)) Whether you are offering Software-as-a-Service (SaaS) web application, Platform-as-a-Service (PaaS) or Infrastructure-as-a-Service (IaaS) – what you are offering is more than just your code – it is your service.

Even if you do not offer a formal service level agreement (SLA) and have a statement in your Terms of Service that you are not liable for anything, your online application or platform is still a service so your customers expect it to be reliable and secure.

At our recent WSO2Con, Chamith Kumarage delivered an excellent session on how our Cloud DevOps team works. If you are delivering a service online (or considering doing so) – make sure to watch the recording (quick registration required).

Here’s my quick summary of Chamith’s advice:

1. Automate everything: repetitive tasks not only are inefficient and mundane, and eat your time. When done manually they are unreliable. Humans tend to do things slightly differently each time they do them, or not do them at all.

2. Tasks are really parts of processes: when you come up with something that needs to be done, ask yourself what is the process flow for this task? For example, a data backup is really a part of a process that includes:

  • Scheduled (e.g. at 1 a.m. every day) script which creates a backup,
  • Some sort of monitoring system which verifies that the script ran and the backup got created,
  • Notifications on failures and procedures that need to be followed in not,
  • Backup testing: automated and/or regular manual recovery drills (if manual then documented and performed by different team members).

3. Design for failure: everything will be failing so make sure that your system can sustain the failures. For example, if your system uses multiple virtual machines in the cloud, keep running a “chaos monkey” script which keeps randomly killing the instances and automated tests which ensure that these instance failures do not affect the overall system (by the way, see how Netflix does that.)

4. Self-healing and success verification are critical for all tasks. Any task and operation can fail (see above) so the system should not get “surprised” but should always automatically validate the action results and if something didn’t go right – implement the healing procedures (start new instances, retry, and so on).

5. Enforce discipline, processes, automation, checklists. Document everything. This will make your processes repeatable and reliable.

Bus monkey test” (related to the above) if one of your team members gets hit by a bus – all operations should keep working: everything needs to be documented and tried by other team members. (* This is a mental experiment – do not actually hit your team-members by busses :))

6. Monitoring and analytics: the key is not to collect and show tons of data and alerts, but be able to quickly detect abnormal behavior.

7. Communications: your dashboards should quickly and clearly give you the big picture and relevant details. Key metrics and system state should be something that everyone sees and understands, effective drill-downs should make it easy to understand and fix stuff.

8. Agile delivery: waterfall processes in the cloud are bad and stressful.The smaller the changes and the more often and in more automated fashion they are – the more mundane they become: which lowers the risks and improves the skills and reliability. Cloud and big-bang releases do not go well together.

9. Use standard tools and native systems of underlying platforms – do not reinvent the wheels. For example, if the platform gives you SQL-as-a-service (Amazon RDS, Azure SQL and so on) – use those and not your own MySQL running on a virtual machine.

10. Post-mortem analysis is a must. If something did get wrong after all, you need a formal investigation process:

  • What happened?
  • Why and what needs to be done to prevent this in the future?
  • If automated monitoring didn’t catch it, why and what needs to be done to prevent this in the future?
  • If validation and self-healing didn’t catch it, why and what needs to be done to prevent this?

Full session recording and slides are available here.

Building WSO2 Cloud

This year I changed my job and became in charge of the Cloud business at WSO2. This means that now on this blog you will have even more “cloud stuff”, so I thought I would start with a quick intro to why I joined WSO2 and our general cloud directions.

Sorry if the stuff below sounds a bit salesy. :)

WSO2 started as a middleware company – with an enterprise service bus – WSO2 ESB – that is still one of the leading products in the industry.

For those who do not know, ESB is basically a system that helps connect all the discrepant IT systems that you might have – all talking different protocols, formats, etc. – so you can create your enterprise applications that use all the building blocks that you have in the company.

What made WSO2 cool (besides amazing performance and the fact that everything they produce is open source under Apache license and that all the core product discussions happen in the open, in public newsgroups) is that early on, the company designed the product to be a platform (called Carbon) which has many building blocks (for identity management, various protocols, event processing, transformations, analytics and so on) which the company then used to deliver a huge set of successful products:

  • ESB,
  • API Manager,
  • Identity Manager,
  • Business Analytics Monitor,
  • Complex Event Processing,
  • Private PaaS,
  • App Factory,
  • Enterprise Mobile Manager,
  • and many more.

All of these built from the same Carbon platform – which is very impressive.

This basically pushed the company from middleware provider to become a supplier of one of the most comprehensive platforms that the biggest enterprises are using to turn their IT platforms into Connected Business – unified system serving APIs, mobile apps, portals, applications and services connecting the company’s resources with employees, partners and customers.

Companies like Boeing, eBay, and StubHub are using this platform to run their systems – so inadvertently you have probably been a user of WSO2 technology one way or another.

Now my goal within WSO2 is to extend the reach of our technology to companies which would like to consume it as a service.

Coming from both enterprise and cloud background, I can attest that the collection of technology that WSO2 has is probably the most comprehensive cloud platform in the industry today.

At the moment, this technology gets used mostly in private cloud scenarios. My team is now actively building the public cloud side of the story. Stay tuned!

Website scraping to PowerShell module

I have not tried this one myself, but I have to admit: this Gargl video on turning Yahoo search site into a PowerShell module looks pretty cool:

See Joe’s blog post for details and download links.

Upvote PowerShell in WindowsITPro Community Awards

WindowsITPro is doing their annual Community Choice Awards voting – and there’s a bunch of PowerShell-related nominees:

1. SDM & SpecOps in Group Policy – both have PowerShell,
17. Best Wireless/Mobile: SAPIEN iPowerShell
20. Scripting Tools: PowerGUI, PowerShell Plus, PowerShell, Sapien PowerShell Studio,
25. Task Automation: Microsoft Windows PowerShell,
26. Training and Certification: Sapien PowerShell videos,
28. Free and Open-Source: PowerShell Plus, PrimalForms.

The voting closes tomorrow – Sept 20, 2013 – so there’s still a chance to upvote the PowerShell tools that you like. :)

PowerShell Script to Bulk-Change Excel File Formats

Today I had to find a way to change file formats for a lot of files. Here at Jelastic we use JIRA for bug-report tracking. Every week I am getting tons of automated reports from JIRA to send around to our partners and so on. The problem is that JIRA exports reports in some weird html format with xls extention, and although Windows Excel can open it (after displaying ‘the file format and extension do not match’ warnings) – Macs cannot.

Opening and resaving lots of files manually was not an option – PowerShell was. :)

[UPDATE] I’ve made a few minor changes per feedback from Stanley – now properly managing file extansions and formats.

Here’s the script that you can reuse if you have a similar problem to solve:

############################
# PowerShell script to open all Excel files in a folder
# and re-save them in proper format
# (c) Dmitry Sotnikov
############################

# create COM object to use Excel
$objExcel = new-object -comobject excel.application

$objExcel.Visible = $True

# open the files to re-save one by one
dir D:\myfolder\*.xls | ForEach-Object {
$doc = $objExcel.WorkBooks.Open($_.FullName)

# Save in new format and with new extension
# Format codes can be found here:
# http://msdn.microsoft.com/en-us/library/office/ff198017.aspx
$doc.SaveAs(“$($_.Directory)\$($_.BaseName).xlsx”,
[Microsoft.Office.Interop.Excel.XlFileFormat]::xlOpenXMLWorkbook)
$doc.close()
}

$objExcel.Quit()
$objExcel = $null

Just Script It!

Here’s Sean’s newest crazy PowerShell video in which he managed to feature your’s truly doing moonwalk (or trying to ;))

As you can see from the video – Microsoft MVP Summits are a lot of fun with amazingly smart and crazy people around.

No more one off IT management. Just Script It!

PowerShell script to set Skype status text to latest blog or twitter update

In my current company (Jelastic) we have something happening to us all the time: latest blog posts, awards, media mentions, etc. We are doing a decent job pushing these to various social media, but I also wanted to get these to my contacts in Skype (Skype gives you the ability to set your status text in your profile).

Below is the PowerShell v3 script that I wrote today to do that! :)

It takes the latest item from my blog and twitter feed, sees which of them is fresher, and (unless the tweet is just my reply to someone) pushes that to Skype (the property is called MoodText).

Here’s the script:

###################################################
# Set-SkypeStatusText.ps1
# Gets latest post from RSS feed (e.g. blog) and Twitter
# Picks whichever is the latest and sets it as status text (MoodText) in Skype
# (unless the latest tweet is a reply)
#
# NOTE: On x64 boxes, use PowerShell x86 (for Skype compat)
#
##################################################
# (c) 2012 - Dmitry Sotnikov
##################################################

# Customize these for yourself
$myblog = "http://blog.jelastic.com/feed/"
$myTwitterHandle = "DSotnikov"

# Get the blog feed
$blogFeed = Invoke-RestMethod $myblog

# Get the twitter feed
$twitterFeed = Invoke-RestMethod `
"https://api.twitter.com/1/statuses/user_timeline.rss?screen_name=$myTwitterHandle"


# If twitter is more recent and not a reply (does not start with @) use it
if (($twitterFeed[0].pubDate -gt $blogFeed[0].pubDate) -and
($twitterFeed[0].description[$myTwitterHandle.Length+2] -ne "@")){
$latestPost = $twitterFeed[0].description.Substring($myTwitterHandle.Length+2)
} else {
$latestPost = "$($blogFeed.Item(0).title): $($blogFeed.Item(0).link)"
}

# Set the status in twitter
$skype = New-Object -ComObject Skype4Com.Skype
$skype.CurrentUserProfile.MoodText = $latestPost

Now if you want to have this happen automatically you can just schedule it using Windows Task Scheduler.

Important:

  • Make sure that you use 32-bit (x86) version of PowerShell if you are on 64-bit Windows – otherwise Skype object will not get found (so the filepath for the Windows task on x64 Windows will likely be %SystemRoot%\syswow64\WindowsPowerShell\v1.0\powershell.exe)
  • Either sign your script or set ExecutionPolicy to RemoteSigned – otherwise the script execution will fail.

My Recent Tweets

Legal

The posts on this blog are provided “as is” with no warranties and confer no rights. The opinions expressed on this site are mine and mine alone, and do not necessarily represent those of my employer - WSO2 or anyone else for that matter. All trademarks acknowledged.

© 2007-2014 Dmitry Sotnikov

April 2014
M T W T F S S
« Mar    
 123456
78910111213
14151617181920
21222324252627
282930  

Follow

Get every new post delivered to your Inbox.

Join 93 other followers

%d bloggers like this: