Monday, December 28, 2009

How Operational Data Store Works

The operating system (OS) is the first thing loaded onto the computer -- without the operating system, our computer is useless. An Operational Data Store - ODS is an integrated database used for being an interim area for a data warehouse.


We have several database type systems:- transactional- operational, data warehouses etc.

If you think about how our brains work the three databases above would equate to the following:
- transactional = doing something and thinking about it;
- operational = using your short term memory to remember what you did a few days ago;
- data warehouse = using your long term memory to remember what you did 10 years ago.

It is common to use the different databases for various storage uses. The information keeps go through a life cycle, just like we do. And various databases or data structures are used for example to house the banks information at the appropriate life cycle.

Here's an example
Let's say that at the bank there is a team that takes orders for the Bankers' deals. This team needs a database solution for storing all the incoming Banking deals. On a transactional level, this team needs to store the- deal description- deal time- deal due date- deal amount Usually lots of data. This team will also connect to other database tables that store things like:
- customer (in terms of companies)
- person ( in terms of a companies employees)
- meetings (set up at customers for the investment bankers to meet with the employees)
- bank employees (need to store which bankers are going to meet with what customers and are assigned to which deals).
Many of the above tables are now transactional, they are updated everyday. The updates can be inserts of new records, updates of records, and even simply reading the records for viewing on a computer screen. Now there's a cost associated with building out data centers in terms of database software, server software, load balancing, failovers, disaster recovery and the way the data is used will dictate how a data architect will solution for the hardware, physical and logical architecture of the data solution. At a certain point, the team helping the Bankers, some of those deals go from the really fresh world of first inputting and make some expected updates to the bankers meetings and deals.


Then a data migration takes place, where rows of data are pulled from the transactional database and moved to the operational data store. In the staging area of the operational data store the normalization of the data will change. There's all sorts of steps andrules here. Basically, the architect will not change the data structure too much, but some, because of the type of reports that will be pulled from the operational data store, are more exhaustive than those from the transactional database.
There's a lot of theory aboutatomicity, data scrubbing, normalization, but it all basically relates to getting all the data in, setting it up in tables with relationships and likely building out a much of the logic that will drive reports via triggers, stored procedures and actual programs.
Some of the above is about dividing up the work between the application servers, the database servers, and tuning the application so that when the Bankers team needs to pull up a report from the last month about all of the meetings the bankers had, the team gets the report quickly.
Eventually, the data from the ODS goes into a data warehouse where even more normalization goes on and data marts are built off of the warehouse so that historical reports, trends analysis can be performed on the banks business trends.

These type of warehouses generate business intelligence reports by allowing for graphical reports to be drilled into, deeper and deeper so that researchers at the bank can understand the bank, customers, world economies, etc. An important thing to remember is that there are more transactional databases than ODS. And there are more ODS than data warehouses. The further one gets from the transactional database, you typically look for different hardware, software and even storage facilities.

Sunday, December 20, 2009

ETL architecture

To describe ETL architecture we have to learn first of all about short history of information specially in companies were certain data created in some certain way is needed to succeed.
At the beginning was a real mess with data which was tried to implement into the people’s minds and everybody around let’s say “big bosses” was expecting from those poor people to use that information in correct way, to know everything about information which were circulating within the company. It was real misunderstanding from side of those people who should be run companies in certain way which would allowed to maintain good atmosphere. among everybody within the firm.
Fortunately some really smart people in form of academic scientists, managing directors mainly on business ground made some struggle and created very very good tools which caused revolution in the area of managing of information.
Because it was plenty of ideas and plenty good tools so every firm has its own choice and it can choose from the verity of tools which are not replaceable in today’s world.

Let’s write something about one of them as I mentioned before. ETL architecture which allows to extract, transform and load data into certain way which allows to use that data in normal mode for example to sell things from warehouse of certain company by people employed there which they have not knowledge about specific and all data which are in that firm because they do not need to know. It is enough for them just this particular we can say computer program created for that purpose which allows them to work smoothly and reach goals set up by management.

To describe and give the certain definition to ETL architecture we can say that it is the core process of data integration and it is mainly connected and linked with warehouse data. ETL is able to extract data from the source let’s say from the top to the new format which is needed for particular business rules in the company and then set up as a target structure for that firm.
We see here that ETL architecture is the tool which is so much needed in today’s world of companies where certain good planning and managing of information by these tools not necessary ETL architecture because we have plenty more of them can be really the best way to the success.

We can divde this particular process of transforming the way that data works for us.
First of all we can use the ETL architecture in more simple way to reach target point but that will not be working at the highest level of our expectancy. That is why we can use more complicated way to use the same system to transform our data but in other available way and when we finish we will have program in which will be able to work more effectively because as it is in the live easy ways to do things something are not necessary good in longer terms. So we have to sometimes choose ways where is the more work throughout the project and at the end of it we can enjoy program with whom we can work easier and our work is more effective, simpler and bring some good results.

As we can notice that all tools are used very often nowadays in business where we need to have certain information, we can say while the client is waiting, very fast plus must be accurate and reliable.