How Big Data can help in meeting big energy needs

Big Data makes a big difference in how we generate and distribute energy and forecast our energy needs. Let’s take a closer look at how the energy sector is levaraging Big Data to ensure cost-effective energy generation.
The world’s demand for energy is insatiable whether it is solar, wind, tidal, oil and gas, thermal, hydro-electric, wood, nuclear energy.
There are two types of energy — renewable and non-renewable. The non-renewable energy is scarce such as fossil fuels and nuclear material and is found in earth and there is a limited availability. This energy we use in our daily lives, whether in our automobiles or for generating electricity, which powers our home.
Renewable resources regenerate as we use them and include solar, wind, water, magnetic and bio-fuel created from farming. This is the future of energy for the planet and its inhabitants and is generally known as green energy.
As the cost of energy increases and its availability decreases there is an extensive use of collating data in the discovery, extraction, processing and transmission and distribution of energy.
The energy business is increasingly using Big Data and cloud computing to ensure efficiency and cost effective solutions. Let’s take a closer look at the role Big Data and cloud computing is playing in the energy sector.
Exploration:
We will begin with exploration — a geologist needs to identify the depth at which the non-renewable fuel can be found, the expected quantum that can be extracted and also the quality and yield. The geologist on the field collects data of a number of exploratory sites to create humongous quantum of data. This Big Data has then to be sieved and mapped into economics, local conditions and then a choice has to be made on which locations will be explored. This is an exacting and back breaking activity and on this rides millions of dollars of profit or loss of energy exploration companies. Big Data is able to handle discrete and non-discrete data and provide significant results in identifying potential sites for exploration.
Extraction
Extraction is a challenging task and has environmental issues to it, both in the harshness of the locations where the energy is extracted plus the impact of environmental damage caused as a by-product of extraction. The regulatory laws are getting tougher both for managing the people who are at risk at these extraction sites, which are a high risk environment and also in case environmental damage is caused. Big Data helps the extraction companies manage these risks and also arrive at optimum risk and ensure governance and best practices to contain risk. Many exploration companies use Big Data to manage a number of extraction by automated systems and keep the need for humans in the sites as low as possible.
Processing
The extracted fuel has to be processed to get the energy in a form that can be used, be it petrol, diesel, gas, shale, coal. The processing again needs data management to arrive at optimum output from the given raw material. Big Data is also extensively used in processing.
Generation 
When energy is converted from one form into another, mostly into electrical energy, Big Data computation and analytics again plays an important role in generating this energy and optimizing the generation needs to bring about efficiency and ensure cost-effective energy generation.
Distribution 
Distribution of energy and the supply chain logistics also has Big Data in the forefront. Whether it is electrical transmission and distribution grid management, oil pipes for pump liquid fuels or managing wind and tidal energy firms, cloud computing and Big Data is used to reduce the distribution losses and deliver at the most cost effective methods.
Metering 
Metering of consumption and prediction of usage of energy is important, one for collection of revenue and second for forecasting the energy consumption.  Also, green buildings can be monitored and managed by using applications in the cloud, for that matter city energy grids can be managed and optimized by using applications in the cloud coupled with Big Data.
Energy Markets
The energy markets are growing globally and cloud computing and Big Data analytics today plays an instrumental role in helping traders make informed decisions in the market related to risks and making profits from energy trading.
The future of clean, efficient and cost effective energy is being driven by the use of Big Data and cloud computing.
 Cloud is by itself a significant contributor to green energy and contributes in reducing the carbon footprint of the world.
With Cloud and Big Data we can take it to the next level by providing clean, renewable and cost effective energy for the planet.

This was published in Information Week – India.
http://www.informationweek.in/software/13-05-03/how_big_data_can_help_in_meeting_big_energy_needs.aspx

The Next Gold Rush in Financial Markets will be driven by Big Data

The next gold rush in Financial Markets will be driven by Big Data, the key driver for this is the low cost to create, manage, mine and deploy big data using the pay as you use  solutions in the cloud.
Cloud computing has democratized Big Data usage and analytics, the differentiator for the winners will be the traders who use Big Data Analytics and Algorithmic trading engines to churn profits from the markets.
All markets work on information whether it is from government, corporate, industry, social media, economical trends, sales figures, people, historical statistics and future projections and more.
All this data both structured and unstructured will have to be brought together under one umbrella and made to work in tandem to provide analytics which can be piped into an algorithmic  engine which in turn will create the upside in the markets for the bold investors who get this technology usage right.
Big Data will allow traders to harness the energy in information to create value in the financial markets.
Here is a scenario on how could this happen,  let us consider the US Bond market trader who wants to use this combination of big data analytics and an algorithmic engine , wall street data over the years, data on government spending, SEC guidelines, the Federals positions, the various bond gurus prophesies, the news both digital and printed, plus the market prices and the historic data, this list is not complete and there are more pieces in this big puzzle including the various trading strategies that will need to be adopted.
The trader will have an almost infinite information set which needs to be mined with the right tools and analyzed to get the winning formula; we will need some of the best mathematicians and statisticians in our team to prioritize and map this data into the trading strategy and predicted outcome.
The trader will need Big Data to hold the humongous volume of structured and unstructured data and  his analysts will need the ability to toss and turn the data . The trader will need a team to help managing this jumbo canister of digital data.
There will be a need for a  team to scrub and clean this data before it can be used by the customized tools for analysing this data. The team has to be adept at building and deploying tools to clean and scrub data in real time.
There will be static data streams and real time data streams which have to be merged by the big data algorithms to give the trader an useful output to feed the algorithmic trading engine which will connect into the digital bonds market.
Remember all this activity of data  production in the real market will happen in nano seconds or less in a fast moving and frenetic digital market and will be more complex in the equity and commodity markets which are like quicksilver.
If the trader decided on hedging, cross border correlation with other markets and also within the domestic markets the data flows and information engines become more complex but the upside for the winner will be take it all.
How do we put a team together to make this happen, in  my mind it will be a global project,
Big data and algo engines are best today in the USA so we will use the computing infrastructure in that country and also work in the markets in the USA .
Asia  could be the ideal location for data cleaning and preparation, it has a competitive and educated work force which have shown that it can deliver, India could be a leader in this space given its Y2k migration track record. But other countries like China, Korea are also excellent sources.
The algorithmic engines complemented with  mathematics and statistical engines may come from the team in Eastern Europe who have wonderful teams who have made significant contributions in this area.
India could provide the expertise in building the solutions need for data analytics and algo trading engines and other related technology engineering.
The funding for this gold rush would not be from the European nobility but from ordinary citizens who believe in the power of Big Data and its positive outcome.
Once we have the ingredients together and we will need this team of Americans, Asians, Europeans to be connected virtually to make our gold mining expedition a reality and to take on the global markets.
What will be the time needed for such a project, need not be more than three months if we use cloud computing project management methodology. We can start small and keep improving our big data analytical engine and our algo trading engine as we see an upside in our technology.
The downside could be we lose our capital if this does not work, but we have a small investment to make on cloud computing, but with the right team there will be no looking back.
The biggest worry in this strategy is whether the global exchanges will be able to handle the tsunami created by the marriage of big data analytics and algo trading. The barrier  for this project  could be the exchanges and regulators  who may play spoil sport and close this window of opportunity even before the party starts.

The early bird catches the worm, so get your tools sharpened, put your tools together and start your Gold Mining expedition, if you are lucky you may even strike platinum. Markets are all about slices of opportunities like a surfer finding the perfect wave, so get your boards out and ride the waves of this new opportunity. Nerds can change the global markets and Big Data will the technology engine that they can harness  to make the difference



This Post appeared as an article in Information Wee, here is the link .
http://www.informationweek.in/software/13-03-18/the_next_gold_rush_in_financial_markets_will_be_driven_by_big_data.aspx

Big Data – Tread Carefully

Big Data is the hottest technology for 2013, but there is a need for better understanding of Big Data and its outcome.
Big Data is not about Cloud Computing though it uses the cloud as the computing engine.
Big Data it is not a killer technology nor a killer app, rather it is a distilled version of the best apps and solutions to handle huge volumes of data bundled with awesome analytic and graphical tools, which churn out output faster then ever before from Big Data (measured in Terabytes and Petabytes).
You need large quantum of clean and usable data to get a ROI and also effective and useful results working on Big Data,.
The fist step is clean data, the second step is what we want to achieve by analyzing this clean data and for this we need a road map, at least we will have some idea of where to reach, though our destination may change in our journey as we sieve through the Big Data..
The right team will succeed with tenacity , knowledge and ability to handle analyse and interpret the results provided by the Big Data tools, assembling a highly competent and skilled set of people to work as a team would be the challenge.

How many Hues can you get in this picture, sieving through big data needs better skills.

How many corporate own Big Data, does it stop at the fortune 500 companies or does it extend beyond that into companies who own Big Data especially Government, Health care, Banking and Financial Services, Consumer Retail and Telcos and also Politicians  Will companies give their data to smaller niche players who specialize in Big Data analytic and interpretation, higher the compliance  lower will be the probability of Big Data moving out an organization.
Do Big Companies have the Big Data as a Strategic Tool in their Plans? Small companies do not have Big Data so who qualify as Big Data Users?
Well for one big data will be used to protect cyber crime and also be used by cyber criminals.
Big Data tools can be used to segment the world’s demographics on color, beliefs, attitudes and other attributes. Wars  will be won by master minds who can use Big Data to crack genetic codes and use Big Data to fine tune their strategy to Win at any Cost.
It is important to recognize Big Data tools are potent weapons in the wrong hands and the time to curb it is now before it explodes.
Herein Lies the GAP, we need to harness Big Data for the betterment of mankind and not for destroying the world, it needs to be tamed unlike Nuclear Science which today in the wrong hands has the potential to blow up our fragile world.
Big Data is more potent then a Nuclear Bomb and therein lies the GAP and the Paradox of Big Data!
We need to Tread Carefully into Big Data !

Cloud Conferences and Cloud Conversion

It is the season of Cloud conferences spewing sermons on cloud technology in India and rest of the world.
Cloud Computing will complete a decade soon and we still market cloud services like traditional technology.
Cloud is an utility and we need to learn to deal with selling and marketing cloud computing as an utility.
Do electricity and water companies have conferences to market their services, cloud computing business need a fresh sales and marketing team to retain and acquire new customers.

Learn from Religion for Cloud Conversion 

Cloud adoption is there to stay and the next step is bringing in more followers into the cloud computing fold. We need to change from evangelistic mode to Missionary mode and take a leaf in marketing from the church or similar religious organizations and borrow their strategies for conversion.
Does cloud computing needs a conversion strategy borrowed from Religion, I believe it does because the traditional IT leaders are critics of the cloud and still cling to their old ways.
So where do we begin, just like  converting the “king or ruler ” was important for a missionary ; we too need to target the Chairman/CEO and Board of a company for cloud conversion. Talking to the CIO/CTO will get us nowhere in cloud adoption. Once the King is converted the subjects will follow.
So as you make your marketing plans for the next year spend more time taking the cloud into leadership and CXO conferences rather then Cloud preaching  cloud technology to the non believers (CIO/CTO/IT Professionals)..
Conversion will begin in the corner office, and the target for your team for 2013 will be to increase the corporate user base of cloud computing by at least 100 times from your current customers.
Please add this to your annual planning exercise and may cloud computing continue to usher in prosperity  economic benefit, agility and innovation to the cloud service provider and the organizations that benefit from the cloud.
Wishing you’ll a cloudy 2013 in your planning for the next year!

Big Data – A new CBDO may lead rather then the CMO & CIO

As Big Data throws up more patterns and helps companies carve out new markets the battle lines are drawn on who will lead the organization in creation value from use of Big Data.
Will it be the CMO (Chief Marketing Officer) or The CIO (Chief Information Officer) and if they work as a team who will lead the team ?
Big Data may not be useful for all Businesses but those who have large data sets and are looking for patterns in complex market place are those who may benefit the most.

The CMO understands his markets best the CIO understands the technology tools used on Big Data, but there is a vast difference in their approach to problem solving, skill sets and operational styles.
A CIO understand the business operations, the CMO understands the markets best, the CIO delivers the strategic intent with technology as a driver, the CMO delivers real numbers and brings in the cheque to make strategy work.
A difficult decision but any organization will go through a learning curve before the leader is anointed, in some organization the CIO and CMO may lose out on leading the BIG DATA Revolution and a rank outsider who understand Big Data analytics’ may be the ideal choice.
Will there be a new position for the CBDO – Chief Big Data Officer  in the corporate landscape?
We will know soon as Big Data Adoption accelerates and becomes the strategy enabler in organizations.

Bharat Badal at Interop 2012 @ Mumbai on 11 October, 2012.

L S Subramaniam
11 October, 2012 l 11:30 am – 12:00 pm
Topic: Bharat Badal – How the Cloud can transform Rural India
L S Subramaniam
Cloud Evangelist & Senior Advisor Power Exchange of India
Abstract: The ability of the cloud to provide agility, innovation, speed of execution and in reducing the barrier to access the best ICT infrastructure for computing is well known and documented. However, the much bigger impact of cloud computing will be felt in rural India. Cloud computing has the potential to substantially reduce the CAPEX spending on ICT by the Government and allow it to provide the digital highway to bridge rural and urban communities, besides acting as a catalyst to shape a knowledge-driven economy. The cloud allows for an equal field irrespective of location and net worth. If one has an idea and the skills, every citizen will have the same opportunity to leverage on the cloud for adding to the economic might of our great nation. This session will explore the benefits of Cloud Computing to the Indian citizen through a Pan Indian National Cloud – Bharat Badal.

INTEROP Mumbai 2012

Are you ready for the Deluge of Private Clouds !

Public Clouds have shown the way that Cloud Computing is feasible and a disruptive technology which business needs for agility and economy.
In the next  year the private clouds, real private clouds (not virtual data centers named as private clouds) will exceed the public clouds we have today .
Governments across the world will build numerous private clouds for their own national requirements, large corporations especially the Fortune 500 companies  will build their own private clouds ( http://www.fortune500s.net/fortune500-list.php). Large corporations function like governments given their size and reach.

Are you ready for the deluge of Private Clouds!
A number of companies in fortune 500 list already have private clouds and many like GE have launched world class energy efficient green data centers to meet their computing needs. 

Some of the smaller companies will build their private clouds, but will have to take the hybrid cloud route for cloudbursts since they cannot justify the cost.
The next wave in cloud computing will be IT Governance of private clouds which will need third party consulting and  assurance to keep the lights burning.
The number of private clouds will be almost 5 times more then then the public clouds available today and opens a huge opportunity for Cloud Enabled Data Center Engineering. The big winners in this opportunity will be IBM, Dell, Cisco, HP, Microsoft, Vmware and a other companies who partner these IT giants.
In cloud consulting the winners will be IBM, HP, Deloitte, KPMG, Accenture, Cognizant Technology and PWC.
Are you ready for the deluge of Private Clouds !
Are you ready for the next big opportunity in Cloud Computing ! 

It is raining, PDA,cellphones, smartphones, Slates, Tablet PCs & solar netbooks! Let us use them to enlighten the world !

There does not go a day without an announcement of a new device in the marketplace which will connect us to the cloud. The most important drivers of cloud computing are the economical and easy to acquire endpoint devices PDA’s,cellphones, smartphones, Tablets PC’s, slates and Solar notebooks.

This are the real messengers of cloud computing, they are the devices which allow people to exploit the benefits of the cloud be it in a social or business context. These are the tools that we can use to enlightent the world !

No longer do you wait to go home to upload your video of your holidays, they are uploaded real time thanks to these devices. And you may have millions of people all over the world sharing your experiences and learning with you.
Imagine what it can do for education, it can allow us to take the Khan academy http://www.khanacademy.org/  or similar great organizations to millions of knowledge seeking inhabitants of our world.
Imagine delivering the Khan videos in the language of the listener using a cloud based solution for translations, this is not science fiction but can be achieved using  existing computing tools .

Imagine an educated and empowered world, which is captured beautifully by Rabindranath Tagore in his timeless poem :

Where the mind is without fear and the head is held high;
Where knowledge is free;
Where the world has not been broken up into fragments
by narrow domestic walls;
Where the words come out from the depth of truth;
Where tireless striving stretches its arms towards perfection;
Where the clear stream of reason has not lost its way
into the dreary desert sand of dead habit;
Where the mind is led forward by Thee into ever-widening
thought and action–
Into that heaven of freedom, my Father,
let my country awake.

Here please interpret country as One world and a younger world without walls and barriers for learning and sharing knowledge.

Knowledge dissemination does not need two point factor authentication or other security devices, what it needs is an economical engine like cloud computing which can be harnessed to bring the world together as one country and one nation.

We must use  cloud computing to integrate the world  into one country like the steam engine did a few centuries ago for the very survival of our planet.

May may one day the imagination echoed by by John Lennon in his song Imagine will come true for our world with a little help from Cloud Computing !

http://www.youtube.com/watch?v=-b7qaSxuZUg

Cloud Computing – A Powerful tool for cyberattack !

Conventional discussion on cyberwarfare has  dwelt on information warfare, though information warfare is here to stay which was validated by the cyber attack on Estonia in  2007 when the small country was crippled and the country had an emergency.

The Estonian attack seems like simple  influenza when we analyse what the cloud computing services can facilitate as tool for  cyberwarfare ; this would be an attack of  pneumonia or even pleurisy with assured outcome as death.

The key strengths of the cloud are what is every Generals dream for fighting a war:

1. Rapid Elasticity – The cloud will not go down and is designed to withstand huge demands of computing when required. This is like men in an army who never get tired and the number of available fighting men are almost infinite.
2. On Demand Self Service – Anyone need to register a domain and you are fine to use the cloud services, you will also need a legitimate credit card for paying for the services. Both these identities are a breeze for any illegitimate agency whose intentions are malicious. Anyone can play General if he has a valid email id and a valid credit card, then he has access to his cyber army.
3. Resource Pooling – The cloud service provider will support your applications even if it calls for a cloudburst, this is like an army general with infinite ammunition in  conventional warfare.
4.Broad Network Access – You can access the cloud services to mount an attack from any digital device which can access your cloud service provider. The cyber attacker could  sit in a cave and use the cloud services through a satellite phone to execute an attack.
5. Measured Service may be the only tool that may be a deterrent to a cyber terrorist using the cloud since any abnormalities in storage, processing, bandwidth, and active user accounts can be monitored and suitable controls could be initiated. But if monitoring is not real time the General would have finished the cyber war before the measured service cuts off  the computing supply.

The  Cloud Infrastructure as a Service (IaaS) would be the preferred cloud service because it gives the cyber attacker an opportunity to deploy and run any software including Operating systems and applications.  PAAS and SAAS  can also be used as auxiliary armoury for a sustained cyber attack.

The cloud is already being used to break codes, passwords and algorithms because of its awesome computing power. Imagine what will happen in case of a planned cyberwarfare.

The cloud can be used to bring down all forms of communications telephony, wireless, digital, satellite and any other communication like radio and television which depend on information technology. Once the communication is down the utilities like power, transport (air,sea,road & rail), power grids, power stations will be brought down; the list is endless and all this can be done in a short period by mounting a well orchestrated  cyber attack using the power of the cloud.

The cloud is  a two edged sword and is like the nuclear energy which can be a great enabler but also a pain as nuclear arsenal in the wrong hands.

The cloud computing resources can be used to mount an attack on citizens, corporates and governments or even international agencies everyone is vulnerable.

We need to understand the perils of the cloud and strengthen its soft & vulnerable underbelly;  outages of services providers like Amazon, Google and Microsoft should be a priority but regulating the use of the cloud for cyber warfare is far more greater.

We need to work together in securing the cloud; restrict access to the cloud to the bad element, so that we can preserve our digital  world for our children and future generations of mankind.