Reduce Cloud bill by 30-40%

Now a days everybody is fascinated about cloud technologies because of its ease, scalability, availability, cost efficiency, independence, etc. Some of them already using cloud for their organizations, some are moving to cloud or exploring cloud to drive value for their businesses. 

Many public CSP (cloud service providers) like Azure, AWS, Google App Engine uses PAY-AS-GO model i.e. you need to pay whatever you are using. Before cloud era an organizations are investing lot of bucks in hardware for setting up infrastructure in data-centers. Maintaining that infrastructure adds more efforts and cost to that. Cloud provides three service models i.e Infrastructure as a service, platform as a service, software as a service. By using these models you can focus on your business values rather than setting and maintaining infrastructure (CSP does that for you).

While leveraging cloud features and functionality you need to be very cautious about cloud usage and its bills. Nobody wants pay extra right? Below are the simple ways that can helps you reduce cloud cost to great extents.

Choose resource type, size, location carefully

Most of the cloud bill comprise of virtual machine usage. choosing correct virtual machine size, type or location will cut your bill.

• Type

In azure resource has multiple types like basic, standard or premium. These types dictates resources with particular features e.g. virtual machine has basic and standard type, in basic type we are not able to configure auto-scaling and load-balancing functionality for VM and in later we are able to do. Likewise for storage accounts we have standard or premium type. We need to use premium type to use replication feature. 

Select basic or standard type for non production (development/test) instances which doesn't requires load-balancing or auto-scaling features. Selecting correct type saves you around 20-25% of cost.

• Size

Correct estimation of instance size is very important in cloud because cost is directly proportional to VM size you use. Overestimating the instance size for product will adds unnecessary cost to your bill. Monitor the past usage statistics of instances and decide the correct instance sizes.

• Location

Depending on the geography charges of cloud resource various. For example costing of Azure resource in East Asian data-centers are cheaper than USA, Europe data-centers. You can host non production instances like development or test in these location to save some cost. 

Setup Auto-scaling

As we know software or services are not used heavily all the time. For example E-commerce, shopping portals serving average traffic during day time and negligible at night. It have heavy traffic during sales, festivals. Auto-scaling in cloud is nothing but increasing or decreasing number of instances (virtual machines) depending on workload. By setting up auto-scaling functionality to infrastructure serves gives flexibility to serve varying workload along with cost benefits. 

There are two type of auto-scaling present in cloud. You need to identify which one to use depending on software or service nature.

• Scale Out-In

In this case number of instances are increases or decreases depending on active workloads(CPU, Memory, IOPS). It is commonly used and there is no downtime experience during auto-scaling. 

• Scale Up-Down

In this case size of instance are increase or decrease depending on experience of past workload trends. In this we experience some downtime during auto-scaling.

Setup scheduling

Starting and stopping of non production instances (development or test) during office hours saves the cost of 12-14 hrs per day. Again keeping instance off on weekends saves cost more hours. You can use software or scripts to setup the schedule for instances. Many CSP also provides scheduling features.

Monitor underutilized resources

Use cloud monitoring tools or scripts which will identify the underutilized instances depending on some threshold values like,

• CPU Usage < 15%
• Network Usage < 10 MB
• Disk IOPS < 100

Identifying those instances and stopping or downsizing them will save wastage of money.

Use linux instances

In cloud linux instances are always cheaper than Windows instances. For operating system independent software use of linux instances, it will saves some bucks for you. Instances hosting Jenkins, Git, Redmine and many more can be host in Linux instance.

Free subscription, Discounts and others

Every CSP provides free subscriptions, announces discounts on services or resources, you need to constantly explore your CSP providers schemes to save few more bucks. Follow the best practices while implementing the infrastructure, analyse bills every month carefully, I am sure you will find something which will help you to reduce cloud bill.

These are the few tips by which you can reduce cloud expenses. Like these tips??

GO implement these tips in your cloud infrastructure and throw a party to CLOUD TEAM after seeing next month bill. :P

Read More

GUACAMOLE: Open Source Web Based Remote Desktop Gateway

Recently, I was searching for remote desktop software using which I can do the remote or ssh login through browser.

There are so many software like thinVNC, Thinfinity, TightVNC which helps us to do remote desktop. These software uses HTML5, VNC variants like thinVNC, thickVNC, UltraVNC. One problem with these software is that we need to install client on machine which we gonna do RDP or SSH.

While exploring I stumble upon Guacamole(Not Recipe). It is clientless remote desktop gateway. We don't need to install any client on machines. We requires only setup guacamole server on separate machine then you only need to access machines through web browser. 

Yeah, It is open Source.

Guacamole is HTML5 web application that provides access to desktop environments using remote desktop protocols. It supports VNC or RDP protocols. We can do SSH login also using Guacamole.

Architecture

Guacamole Architecture

Guacamole is collection of many things, 

- Guacamole Protocol
- guacd
- Web Application
- VNC Client
- Remote Desktop Gateways


Guacamole Protocol
Web browser or application doesn't understand any RDP or VNC or any other protocols which are supported by Guacamole stack. It communicates with guacamole through Guacamole Protocol. It renders remote display and transfer events.

Guacd
It is the heart of guacamole stack. It acts as mediator between the remote desktop machine and web application. It also communicate with the help of Guacamole protocol.

Web Application
We are interacting with guacamole using web application. It doesn't implement any remote desktop protocol but it relies on guacd. It provides UI (user interface) and authentication functionality to users.

VNC Client
It is nothing but java-script client(java server component) which is used to translates VNC into XML format.

Remote Desktop Gateways
Guacamole not only supports VNC but it also support other protocol like RDP, SSH, Telnet, etc. Guacamole stack supports multiple remote desktop protocols through different gateways.

Best part of Guacamole stack is that we doesn't requires to set RDP or web client on each and every machines as other software does.

References:

• http://guac-dev.org
• http://guac-dev.org/doc/gug

Read More

Start-Stop virtual machines parallelly present in cloud service (Azure IAAS)

Time and cost are the two crucial factors in cloud environment, people are paying for what they are using. I have setup my environment in Azure using Infrastructure-as-a-service (IAAS). Environment consists of 20+ virtual machines, virtual networks. I have wrote powershell scripts which provision environment for me.

While working on cloud environment with high configuration machines, I need to make optimal use of cloud environment to reduce my cloud cost. 

For this we can think of  following approaches

• Schedule start/stop of cloud service to reduce cost
• Parallel operation within cloud service to reduce start/stop time

Initially, I have used Start-AzureService and Stop-AzureService powershell cmdlet for starting and stopping of cloud services and it does job for me.

Observations: 

Whenever, I stop cloud service using Stop-AzureService command it stops service successfully. But it is not deallocating the cloud resources. Technically cloud service is in stopped state but its IP address, VHDs, etc. are still in use.  Azure charge for these cloud services. 

As an alternative, I came across Start-AzureVM and Stop-AzureVM powershell cmdlet which start stops the virtual machine present in cloud service. Stop-AzureVM stops specific virtual machine present in the cloud service and also deallocates its resources.

In my environment i have 20+ virtual machines so starting or stopping these machines takes around 60 minutes (each machine takes around 3 minutes to start/stop). So i need to think of parallel execution to reduce start and stop time. I tried to start or stop the virtual machine in parallel in powershell but was not able to do so. I faced mutual exclusion error while performing operation on different virtual machines within a same cloud service. 

In classic model of virtual machine Azure has some restriction while performing the  parallel operation on cloud resources specifically cloud service, virtual networks, etc. We can perform operation on only one virtual machine within cloud service.

We can also manage Azure cloud through management portal. Whenever we start the cloud service using management portal then Azure starts all virtual machines in parallel within 3-4 minutes. While stopping the cloud service using management portal it stops the service and also deallocates its resources. (Isn't it great?)

While exploring the internal working of management portal i came across that we can also manage Azure through Azure Service Management APIs. I wrote the powershell scripts which manages my cloud service. Using REST API, I am able to start/stop 20+ virtual machines in parallel within 3-4 minutes.

Find the powershell script below which start/stop cloud service with the help of Azure service management APIs. 

manageCloudService.ps1

<# 
.SYNOPSIS 
   Manage cloud service using REST API 
.DESCRIPTION 
   Start and stop VMs present in cloud service in parallel 
.EXAMPLE 
   .\manageCloudService.ps1 -ServiceName "testme-cs" -OperationType "start/stop"  
#>

param(
[string]$serviceName,
[string]$operationType)

$deploymentName = $serviceName

# Select an Azure Subscription for which to report usage data
$subscriptionId = (Get-AzureSubscription -Current).SubscriptionId
# Set Azure AD Tenant for selected Azure Subscription
$adTenant = (Get-AzureSubscription -SubscriptionId $subscriptionId).TenantId

# Set parameter values for Azure AD auth to REST API

# Well-known client ID for Azure PowerShell
$clientId = "1950a258-227b-4e31-a9cf-717495945fc2" 
# Redirect URI for Azure PowerShell
$redirectUri = "urn:ietf:wg:oauth:2.0:oob" 
# Resource URI for REST API
$resourceAppIdURI = "https://management.core.windows.net/" 
# Azure AD Tenant Authority
$authority = "https://login.windows.net/$adTenant"

# Credential object
$userName = "organisation user"
$password = "secretepassword"
$creds = New-Object "Microsoft.IdentityModel.Clients.ActiveDirectory.UserCredential" -ArgumentList $userName,$password

# Create AuthenticationContext tied to Azure AD Tenant
$authContext = New-Object "Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationContext" -ArgumentList $authority
# Acquire token
$authResult = $authContext.AcquireToken($resourceAppIdURI,$clientId,$creds)
# Create Authorization Header
$authHeader = $authResult.CreateAuthorizationHeader()

# Set REST API parameters
$contentType = "text/xml;charset=utf-8"

# Set HTTP request headers to include Authorization header
$requestHeader = @{"Authorization" = $authHeader; "x-ms-version" = "2014-05-01"}

$uri = "https://management.core.windows.net/$subscriptionId/services/hostedservices/$serviceName/deployments/$deploymentName/roles/Operations"

# Get virtual machine Name present in cloud service
$roles = ""
$vmList = Get-AzureVM -ServiceName $serviceName |
%{
    $roles = $roles + "" + $_.Name+ ""
    $vmNameForMonitor = $_.Name  
}  

if ($operationType.ToLower() -eq "start")
{
    $body = @"

  StartRolesOperation
  
  $roles
  

"@
 # Invoke REST API
    Invoke-RestMethod -Uri $uri -Method Post -Headers $requestHeader -Body $body -ContentType $contentType
    # wait for vm to start
    $vm = Get-AzureVM –ServiceName $ServiceName -Name $vmNameForMonitor 
    $vmStatus = $vm.PowerState 
    if (!($vm.PowerState -eq "Started")) {   
        do {   
            Start-Sleep -s 5    
            $vm = Get-AzureVM –ServiceName $ServiceName -Name $VMName 
            $vmStatus = $vm.PowerState   
        }until($vmStatus -eq "Started")   
    }
}
elseif ($operationType.ToLower() -eq "stop")
{
$body = @"

  ShutdownRolesOperation
  
  $roles
  
  StoppedDeallocated

"@
 # Invoke REST API
    Invoke-RestMethod -Uri $uri -Method Post -Headers $requestHeader -Body $body -ContentType $contentType
}

References:

• Azure Resource Management using REST API in powershell
• Service Management using REST api reference
• Start-AzureService
• Stop-AzureService
• Start-AzureVM
• Stop-AzureVM

Hope this script will help you guys to reduce your cloud bill. Suggestion and feedback are welcome.

Read More

Want to Ping Azure VM?

Whenever there is issues with any application or machines, the first thing we do is PING. I mean ping to that machine using IP address or hostname (if name resolution setup in the environment).

In Azure cloud we are not able to ping the virtual machines using IP address from outside environment. Ping uses ICMP protocol to communicate with each other. In Azure ICMP protocol is turned off. Azure firewall doesn't allow any ICMP traffic to go in and out of their infrastructure.

Though you can ping Azure virtual machines which are present in virtual network. For this you need to allow inbound rule for ICMP traffic in firewall. You can also disable the firewall. After this you are able to ping the machines present in virtual network using IP addresses.

Note: You can ping the internal or external IP of virtual machine only from inside not from outside the only.

Read More

Install NGINX as windows service using nssm

nginx [engine x] is popular HTTP and reverse proxy server with lot of features. nginx provides executable files for windows environment. We just need to download and unzip setup file to get started.  The official document says we cannot run nginx as windows service. 

We can use nssm (non sucking service manager) to install nginx as windows service. nssm monitors the running service and will restart it if it dies.

Install nginx  

Download Nginx stable version(nginx-1.8.0) from http://nginx.org/en/download.html

Unzip setup and copy in C:\nginx-1.8.0 directory. 

Install nssm

Download nssm zip file from http://nssm.cc/download 
unzip it in C:\nssm-2.24 directory.

Install nginx as windows service

Run below command to install nginx service

C:\> cd C:\nssm-2.24\win64
C:\> nssm.exe install nginx

It will pop up GUI screen in which we have to provide nginx information. Set the application path to nginx executable present in “C:\nginx-1.8.0”. Explore other options present for recovery, login, dependencies, etc. 

Press install service button to install service

Confirm the service installation in services 
o Click ctl+R  to open RUN prompt
o Type services.msc 

Start nginx service and confirm the installation

You are ready to use nginx server. 

References:

• http://nginx.org/
• http://nssm.cc/

Read More

SERF: In a Nutshell

What is SERF

SERF is open source decentralized, fault-tolerant and highly available solution by Hashicorp used for cluster membership management, failure detection and orchestration.

Serf relies on an efficient and lightweight gossip protocol to communicate with nodes. The Serf agents periodically exchange messages with each other in much the same way that a zombie apocalypse would occur: it starts with one zombie but soon infects everyone. It communicates using both TCP and UDP protocols. It is extremely lightweight and uses low memory footprint(5 to 10 MB). Serf runs on every major platform: Linux, Mac OS X, and Windows.

Serf tries to solves problems related to clustered architectures,

• Cluster Membership Management: Serf maintains list of members present in the cluster. It also provides provision to execute custom handler scripts whenever membership changes. Serf agents exchange messages to each other periodically to check the membership status i.e. joining or leaving of members.
• Failure Detection and Recovery: Serf automatically detects failed nodes within seconds, notifies the rest of the cluster, and executes handler scripts allowing you to handle these events. Serf will attempt to recover failed nodes by reconnecting to them periodically.
• Event Propagation: Serf can broadcast custom events and queries to the cluster. These can be used to trigger deploys, propagate configuration, etc. Events are simply fire-and-forget broadcast, and Serf makes a best effort to deliver messages in the face of offline nodes or network partitions. Queries provide a simple real-time request/response mechanism.

Use Cases

SERF is decentralize solution for cluster management so it solves use-cases that are having decentralized state, masterless architecture, and are completely fault tolerant.

• Webservers and load balancers
• Memcached and Redis clusters
• Triggering deployments
• Updating DNS records
• Simple Observability
• A Building Block for Service Discovery

References

• https://www.serfdom.io
• https://www.serfdom.io/docs/index.html

Read More

Merge GIT Branches Using JENKINS

Continuous Integration and deployment or delivery (CI/CD) pipeline consists of many machines or environments. It has development, production, integration, testing and staging machines. Each machine has separate branch in GIT to store the source code.

In GIT, Development branch has all code from developers which are in development phase. After completing development, a developer pushes his/her code to integration branch. It results into running build scripts, testing scripts to check the integrity of code. Likewise code pushed from integration branch to staging and later production branch.

At each stage of CI/CD pipeline, one has to push code from one branch to another or merge them. As DevOps guy one should not do it manually. Jenkins provides the facility to merge two git branches and push it to remote branch as part of build tasks.

Merging git branch using Jenkins task consists of following steps,

• Install GIT plugin in Jenkins
• Create Jenkins task for branch merging
• Configure Jenkins task

Install GIT plugin

Installing plugins to Jenkins is easy. For installing GIT plugin in your Jenkins instance, refer to the link below.

http://blog.cloudbees.com/2012/03/using-git-with-jenkins.html

Create Jenkins task

Create new Jenkins task which will merge the two GIT branches.

Configure Jenkins task

I have created an automergejenkins Github project. It has four branches master, develop, integration and production.

At this post, I am demonstrating how to configure Jenkins task to merge develop branch to integration.   

• Source code configuration

In source code management section, specify the GIT repository and branch to build. I am building develop branch and merging it to integration branch.

Press Add button to add "Additional Behaviours". It shows different build options. 

Select "Merge before build" option and specify branch to which develop branch is going to merge.

   

If you need to specify any build command after merging of branches you can do it in Build section.

• Post Build Action

After configuring the build option, we have to publish merge result to remote branch. 

In "Post-build Action" section add Git Publisher option which will push the merge results only if build succeed.

When you build this task, develop branch got merge into integration branch. You can build the task manually or configure it to build on the other build task results.

References,

• http://blog.cloudbees.com/2012/03/using-git-with-jenkins.html
• http://www.vogella.com/tutorials/Jenkins/article.html
• https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin

Read More