The Data Warehouse Lab: A step-by-step guide using SSIS and SSAS 2017

I always feel some difficulty for newbies in data warehousing to grasp all theoretical aspects due to the complexity of the tools and the long-running process of data warehousing, which starts by selecting the data source and ends by visualizing the information. Therefore, I designed several examples in my course, and I realized that the examples helped students to understand the overall picture better.

Last year, I realized that if all examples can be defined as a kind of story that students can follow them and see the overall picture, it can help even more, so it was the main motivation behind writing this book.

This book aims to help students and practitioners who are new to data warehousing to start developing a new data warehouse project from scratch. It shows different phases of data warehousing projects through a simple case. So readers can experience the full data warehouse development life-cycle through a simple example step-by-step. The book is written for the novice user, so there is no requirement for previous experience of working with MS SQL Server and other tools. However, it expects readers to know basics of databases like the table, columns, etc.

The book does not aim to teach data warehousing and multi-dimensional design principle, nor play the role of a comprehensive reference book on Microsoft Business Intelligence Toolset. It only intends to help readers to gets a hands-on experience on data warehouse development quickly. It aims to give readers basic understanding and experience, so they become more confident in using reference books and online materials.

The book does not go through the installation of tools that are used in the sample project. The readers need to install the following tools in order to follow the steps, i.e., Microsoft SQL Server Database Engine, Microsoft SQL Server Integration Services (SSIS) 2017 , Microsoft SQL Server Analysis Services (SSAS) 2017, Microsoft SQL Server Management Studio (SSMS), Microsoft Excel, and Microsoft Power BI.

For me, the best time to publish the book would be in January, since I did not have time to do serious proofreading on it. However, my course is going to start soon, so I decided to publish the book, and I will correct typos and other problems later. So, please let me know if you read the book and need some help or find some typos. At least, this is the good feature that is provided by Amazon that I can update the content of the book later :-)

Here are the links to the book:
The Data Warehouse Lab: A step-by-step guide using SSIS and SSAS 2017 Kindle Edition

The Data Warehouse Lab: A step-by-step guide using SSIS and SSAS 2017 Paperback

Pivot Query

I always love SQL, and it is really pity that I have not written any complex SQL for really long time (I remember I wrote the last complex SQL statement in 2007). Today, I needed a sort of pivot result from SQL Server. Although I knew that I can retrieve the result very easily using pivot in Excel, I wanted to do that in SQL. Sometimes, I like to suffer myself ;)

First, I realized that there is a feature added to the SQL Server 2008 that I was not aware of, called PIVOT and UNPIVOT! It is really nice feature added to T-SQL, yet it was also possible to do such queries in previous SQL servers with a little more difficulties. However, there is a problem that I discuss through definition of table:

The table has three columns, i.e. userID, movieID and rate. We want to have a pivot view which contains movie as rows, users as columns and count of rates as cells. However, in current cyntax, you have to explicitly mention each user by value, which seems not suitable. Thus, I change a SQL which ‘chandra sekhar pathivada’ mentions in 1/17/2013 in the mentioned page to retrieve the result (look at http://technet.microsoft.com/en-us/library/ms177410(v=sql.105).aspx). The SQL that I wrote is:

DECLARE @columns NVARCHAR(1000) SELECT @columns = COALESCE(@columns + ',[' + cast(userID as nvarchar) + ']', '[' + cast(userID as nvarchar)+ ']') FROM(   Select distinct userID   From movies      ) CLMNS DECLARE @query nVARCHAR(4000)            SET @query = '      Select movieID,         '+ @columns +'     FROM        (        select movieID, userID, rate        from         movies ) x        PIVOT        (count(rate) FOR userid IN('+ @columns +')) AS MyPivot'       print @columns  print @query EXECUTE(@query)

This SQL retrieves the result correctly, yet I am still not satisfied. Do you think that it might be other better way to do that? I am still thinking about it.

The SSAS Tutorial for beginners

It was last summer that I developed a new tutorial for SSAS. The aim of this tutorial is to make students familiar with SQL Server 2012 Analysis Services. It might be helpful for you if you are an absolute beginner; otherwise, there is nothing new that you can learn from. The tutorial is based on a fictitious scenario which is grounded on simplified version of AdventureWorks Database.
Unfortunately, I cannot share the data source of this tutorial.

The olap tutorial 2012 from Amin Jalali

Stanford Machine Learning course

It has been always a true challenge for me to understand Data Mining in depth.

I have always problem to have a deep understanding of data mining algorithm, since all of them were ended to the mathematical formula that I have no idea about them.

Some times ago, I found the Machine Learning course at Stanford University. This is a free course, and I strongly recommend this course to ones who like to have a deep understanding of these algorithms.

This course will give you deep knowledge and understanding about different algorithm. it also gives you clue to distinguish which approach suits you the most in applying Data mining.

Finally, it is really awesome, since you should write some linear algebra programs that implement these algorithms. Therefore, it is not just theoretical, but also it is really practical.

I finish the course successfully, and the course will be repeated in January.

Other courses which are offered free are:

• Machine Learning
• The Lean Launchpad
• Technology Entrepreneurship
• Anatomy
• Making Green Buildings
• Information Theory
• Model Thinking
• Computer Science
• Software Engineering for Software as a Service
• Human-Computer Interaction
• Natural Language Processing
• Game Theory
• Probabilistic Graphical Models
• Cryptography
• Design and Analysis of Algorithms I
• Computer Security

These courses are offered by Stanford University and Berkley University.

MDX and DAX Formatter

Today, I watched a video about "Format MDX&DAX" online application. This application enables you to convert the MDX or DAX expressions into a readable expression. it would be usefull specially when you are tracing your application and using "Microsoft SQL Server Profiler".
You can see the video here.

link of application

BIDS Helper

I participate at Data Warehouse course as the teacher assistant this term.
I face lots of interesting questions and solutions by different students' groups when they asked questions for designing the solution for their assignment.
One interesting things that I have not seen before, was Dimension Health Check facility.

You can install the BIDS Helper adds. It provides you some new facilities that make your job as Data warehouse designer or developer easier.
You can see its facilities and find its link to download here:
http://bidshelper.codeplex.com

Process Mining

Extracting relevant knowledge for decision makers is the key activity in all Business Intelligence areas. Different areas provide capabilities with which we can extract different knowledge from our data. For example, we can discover patterns, clusters and etc. using data mining. An example of pattern mining techniques is sequence mining, where we aim to discover the sequential order of events that happens in a data. Despite the wide use of sequence mining in different areas like biology, these algorithms have not been employed much to add values to business domains. The reason might be the complexity that exists in businesses and the way that we handle our businesses. Many businesses are running through a formal or informal agenda, called business processes. These processes define how different activities in a business process should be handled to fulfill the goal of the business. The relation between these activities can be quite complex, so the analysis of their data could be quite challenging. Let's see the situation with some example.

Imagine that we have a company consists of several informal business processes. This means that we have not modeled our processes, and people know it by heart. If we want to define the business process, we can interview different people who are involved. This is very costly, and it can be biased based on the information that people give to us. We should always consider the probability that not all people tell the way that they work; instead, many might tell the way that they should work! Although we do not have formal business process models, we have the result of execution of our business in different Information Systems. This includes different databases that record different activities, or different log files that persist different actions through time. Business Process Discovery is a sub-area of the process mining that aims to discover business process models through these information. It offers a different algorithm that enables us to discover these models from captured information.

Interesting? Yes! but it is not the end of the story!

Imagine that you have a formal business process model, and the information that records the activities that happened in your business. How can you make sure if what is happening in your business is complying with what you have defined in business process model? Is there any fraud case? Is there any employee who does not know the work but (s)he doesn't know! These sort of questions can be answered if we are able to compare our process model with the information that has been captured in the log files. This is another area of Process Mining which is called Conformance checking.

Wow! so far so good! Can we expect more from Process Mining?

Off course! business processes can capture many different perspectives. For example, they can be so basic that only describes which activity should be performed when! but they can be extended to explain who should perform each activity! Consider a company that has a basic process model. The company might not be able to define who should perform each task at beginning. Instead, the manager lets people work, and after a while (s)he wants to assign people to different activities based on the successful experiences of running the business process. I am sure you are sharp enough to realize that this information are already captured in our databases and log files! so, can we give the basic version of the process model and our log file to an algorithm and expect to receive a evolved version of the business process capturing who should do each activity? Off course! It is called Process Enhancement!

That is amazing! Can we be even more greedy to expect even more?

Sure! The good news is that we can combine Process Mining with other Mining techniques like Rule Mining and etc to expand the power of our magic! If you are interested to know more, there is a research group at the Eindhoven University of Technology that conducts this project, you can find more information on their site.

Today is the third year of this blog! I will get the data mining course next term, and I am very eager to learn and perform it in different contexts.

The Kimball Group Reader: Relentlessly Practical Tools for Data Warehousing and Business Intelligence

If you followed kimball articles, you may be interested to have all of them in an organized collection like a book.
Yes, Kimball published a new book based on these articles. I just could say that it is more than a collection of articles. Indeed, they provide a good description and tips about the whole data warehouse lifecycle using these articles.
I just took a look at it, and it was really facinate me to read it.

IBM Cognos 8 Report Studio Cookbook

I received a free copy of this book in order to write my standpoint on it.

I think if you are new to cognos, or if you worked with Microsoft SQL Server Reporting Services, and want to migrate to cognos this book is very nice to follow.

Indeed, it makes the learning of cognos easier by providing lots of examples which are describes with lots of pictures.

Honestly, I think if you are professional in one reporting tools, you don't need any book, and you can just surf the tool and follow the help to overcome its difficulties.

By the way, if you are not a guru on any reporting tools, or if you do not have enough time to spend on the cognos to migrate from SSRS, I strongly recommend this book.

This book is not going to introduce any concept or any sophesticated techniques on reporting. However, it provides easy to follow step by step instructions to get familiar with the IBM Cognos 8 Report Studio.

Book Link

Solving previous problem

If we analyze the previous problem, we could consider that there are three decisions that the decision maker could take:

1. No treatment
   
2. Treatment with aspirin
   
3. Treatment with warfarin

So, we should draw a decision three with these three decisions at first branch of tree.
There are three different uncertainties that could appear for treatments. Tow of these three is applied also for no treatment.
If we draw the decision three it should looks like the following pictures. (I just provide two pictures, because the tree for treatment with warfarin is the same as the aspirin, just different valuse)






If we calculate the Expected Value (EV, which clarifies the best decision that we could make) it will be clear that patients should be treated with warfarin.

A decision tree of the problem is given at the end of this document.
No treatment & CVA & affected = ‐158,000 SEK
No treatment & CVA & unaffected = ‐76,600 SEK
No treatment & No CVA = 0
Aspirin/Warfarin & side‐effect & CVA & affected = ‐(15,000+158,000) = ‐173,000 SEK
Aspirin/Warfarin & side‐effect & CVA & unaffected = ‐(15,000+76,600) = ‐91,600 SEK
Aspirin/Warfarin & side‐effect & haemorrhage = ‐(15,000+32,000) = ‐47,000 SEK
Aspirin/Warfarin & no side‐effect & CVA & affected = ‐(15,000+158,000) = ‐173,000 SEK
Aspirin/Warfarin & no side‐effect & CVA & unaffected = ‐(15,000+76,600) = ‐91,600 SEK
Aspirin/Warfarin & no side‐effect & No CVA = ‐15,000 SEK
Folding back the tree:
EMV(no treatment) =( 0.25*(‐158,000) + 0.75*(‐76,600))*0.8 = ‐77,560
EMV(aspirin treatment) =(((( 0.25*(‐173,000)+0.75*(‐91,600))*0.644)+0.356*(‐47,000))*0.048) + (((0.25*(‐173,000)+0.75*(‐91,600))*0.5)+0.5*(‐15,000))*0.952 = ‐64,692
EMV(warfarin treatment) = (((0.25*(‐173,000)+0.75*(‐91,600))*0.5)+0.5*(‐47,000))*0.072 + (((0.25*(‐173,000)+0.75*(‐91,600))*0.1)+0.9*(‐15,000))*0.928 = ‐28,639
Thus, if we merely look at the costs for the hospital, patients should be treated with warfarin.

These kind of analysis are called subjective analysis. If you are interested to deal with these kind of problem, I recommend the Making Hard Decisions: An Introduction to Decision Analysis book by Robert T. Clemen.

Decision Tree

I hade a course previous quarter. It was about how to use decision tree for taking correct decisions. It was very amazing and I want to rewrite one of the question of final exam for showing the types of question that decision tree could help us to solve. This quarter I have the second and advance version of that course called Decision and Risk Analysis, second course.

You are working as a medical advisor at a big hospital and your department is specialized within cardiology. Your manager has asked you to formally analyze the medical condition of atrial fibrillation. In the analysis you will merely look at costs for the hospital and not consider other criteria. Atrial fibrillation is a common condition, which carries with it a significant risk for stroke (80%) if left untreated. Treatment with the medicines warfarin or aspirin significantly reduces this risk, but there are side-effects to both treatments. For the average patient (at moderate risk), treatment with aspirin has a slightly smaller risk for side-effects than warfarin, 4.8% as opposed to 7.2%, but aspirin reduces the risk for stroke less effectively than warfarin. The serious side-effects are either cerebrovascular accident, CVA during the treatment or haemorrhage. Of patients treated with aspirin and affected by side-effects, 64.4% get during their treatment, whereas this number is only 50% for patients treated with warfarin and affected by side-effects.

For the patients who do not suffer from immediate side-effects during the aspirin treatment, the risk for CVA is still 50%, whereas the risk for CVA is only 10% for patients treated by warfarin and not suffering from side-effects during the period of treatment. If an average patient (moderate risk) with atrial fibrillation gets a stroke (CVA), regardless of treatment or no treatment, he or she is classified as affected or unaffected. Out of all CVA:s that occur, 25% are affected and 75% are unaffected. Transition costs(those that happen just once) for the treatment of a patient with CVA is established to 76,600 SEK (unaffected patients), whereas the state costs (those that remain patients for their Iifetime) for treatment of a patient with CVA is estimated to 158,000SEK (affected patients). The treatment of a haemorrhage is estimated to 32,000 SEK, and the cost of a medical treatment (either aspirin or warfarin) is 15,000 SEK.

What would be your recommendation to the hospital in the handling of moderate risk patients with atrial fibrillation and why?

But how could we solve it?

If you draw the decision tree, you will see that the warfarin is the best medical treatment.

Predictable Changes with Multiple Version Overlays

There is a technique in Kimball's "Data Warehouse toolkit 2nd" book that is suggested for Predictable Changes with Multiple Version Overlays situations. It was a bit intangible for me what Kimball means for that solution, but I could understand the technique when I give another example to myself ;)

Imagine that we offer variety of products with different categories. Each product could just be belong to one category, and the category of products could be changed at the beginning of each year. However, the category of products could be changed by CEO over the time. It is ok if we want to use Slowly Changing Dimension 2 (SCD2) for this example, but the business requirements that I mention as the following make it impossible to consider this dimension as SCD2.

Imagine we need to answer these questions:

1. What is the sale of a category over a period of time? (This is not the case that makes it impossible to consider the dimension as SCD2)
2. What is the sale of the selected category in question one on the next two years? (This is a real trouble)

It may be a bit vague, so I am going to explain second question. Imagine the sales of category 1 in 2002 is X. Analysts may be interested to know what is the sale of category 1, with the products at the time of question 1, in another period of time. We should consider this fact that some category of products is changed over time; hence, how could it be possible to reflect this information.

This is the situation that we must follow what Kimball says. Consider a column for category of each year in product dimension table.

Product D
------------------
ProductID
…
Category 2001
Category 2002
Category 2003
…

I hope I could explain the situation well. I am eager to know what you think about this technique as well ;)

Second Year Blogging Anniversary

It has been always one of my concerns that I haven’t written any post here since May 14, 2009. The second year of my blog’s lifetime has only four posts that I am very sorry for that. It was occurred because I was too busy with my job and with handling application process of some universities simultaneously.
I want to start my master degree in this year to expand my knowledge about Business Intelligence. Therefore, I resigned from my job to go to Sweden. I think I will write more posts in the following year, and I hope to have wonderful experience about living in another country. I will write about the program soon ;)

The Data Model Resource Book volume 3

Sometimes I spent lots of time for finding the best way to design an ideal data model that could be comprehensive enough to cover all the prospective needs. But you couldn't always do your best, because your knowledge is depends on our experience.
It was previous week that I was looking at lots of books in Tehran Book fair. When I was looking at Wiley's books, a book grabbed my attention suddenly. It looked like a book that I have always been looking for. I believe The Data Model Resource Book volume 3 is a must book that a person who is in charge of database designing or data warehouse designing must have read.
I strongly recommend this book to all my friends who love database designing and data warehousing.
See table of contents and more details about it at Amazon.

Business Intelligence 2.0

Business Intelligence which consists of Dashboard Systems and Decision Support Systems has grown up during long time. Most systems that BI developers have been developed are belonging to Dashboard Systems which are used for demonstrating KPIs and revealing what has happened before. Decision Support Systems which are very depends on using Data Mining advantages for being implemented, have developed much fewer than Dashboard Systems.

BI 2.0 not only focuses on these systems, but it also changes the ETL way that we used to. Developing more Service oriented Applications, and the fact of having service-oriented data sources will change the way that we are using for gathering and cleaning data. Moreover, the needs of considering Business Processes and Business Rules, which exist in BPEL and BRMS, for performing ETL process would change this industry a lot.

I think the next generation of BI has a lot of challenges which are very interesting to study.

Aggregations

Designing appropriate aggregations in SQL Server Analysis Services is a big issue that a BI developer must consider carefully. When I read Performance Optimization chapter of Professional SQL Server Analysis Services 2005 with MDX, I felt that I should study designing aggregation in more detail if I wanted to get a good performance on Enterprise Data Warehouse.

Although "Microsoft SQL Server 2005 Analysis Services Performance Guide" has been published in February 2007, I find it today! It describes some important tips in detail.

If you haven't read it, don't miss it.

Change data capture

Identifying and extracting changed data from source tables was replete with lots of efforts; it also requires designing the databases in an appropriate manner for keeping the changed data within tables or log files. The concept of “Change data capture”, a set of software design patterns used to determine (and track) the data that has changed in a database, helps designers to overcome the difficulty of solving such problems.

Fortunately, SQL Server 2008 implemented a mechanism to enable DBAs for automating this process like what we could consider in Oracle 9i which is called “Capturing Change”.
By using this new feature we could specify a log table in which the changes in data could be tracked. These changes are insert, update, and delete. SQL Server 2008 handles the required operation to store the appropriate data.

The following links are very helpful for understanding this new feature which could make ETL process more easily.

http://en.wikipedia.org/wiki/Change_data_capture
http://msdn.microsoft.com/en-us/library/cc645937(printer).aspx
http://www.oracle.com/technology/oramag/oracle/03-nov/o63tech_bi.html

One Year Blogging Anniversary

It was last august that I started writing within this blog. I started blogging just for making notes on all thinks that I learn about Business Intelligence. I think it is a good idea to write about what we are learning and doing because it makes it possible for us to judge about ourselves and compare each year efforts with other years.
Although it played the role of technical diary for me, it led me to get familiar with a lot of friends such as Pedro, Ella, and others. It was definitely the best experience that I had because I could find friends, and expand my knowledge and consider others’ opinions and roadmap.
I want to apply for a master degree this year, and I hope that I can find a proper master degree. I get the IELTS, and I am currently search for universities while I am contacting with their teachers for perceiving more about their courses. Hard Working and all these activities make writing a bit difficult for me, but I will try to continue blogging, especially now that SQL Server 2008 has released ;)
I am very eager to study about accounting, cash-flow, budgeting, ERP, supply chain management, and etc because I think an in-depth knowledge of these topics could help me to analyze KPIs better.

Thanks you

Best wishes,
Amin Jalali

Microsoft DreamSpark

Microsoft created this site for students in order to access the Professional Developer and Designer tools at no charge.It is very helpful that you can download and use softwares like "Microsoft Visual Studio 2008 Professional Edition", "Microsoft SQL Server 2005 Developer Edition", and etc at no cost.

https://downloads.channel8.msdn.com

Market Basket Analysis

Some time ago, one of the Accounting Division’s staff asked me to reveal if the last decision of the Bank about introducing new participation papers were successful or not; also, he asked me to specify what percent of its sales were because of changing other deposits.
It was a hard job, because we had not any clear idea on how to deal with this matter! It had done, but I kept thinking about solving this problem via Business Intelligence Solutions rather than executing complex queries on databases.
When I was reading “The Data Warehouse Toolkit Second Edition”, I realized that it is very similar to “Market Basket Analysis” concept, and after doing some search it began to sink in.
Indeed, I found it out that it could be done via association algorithm of the Data Mining. This subject is about the market basket analysis, and finding the association of products based on their sales, in order to offer new discount for increasing the sales.
For instance, you can imagine that when you find it out that the toothpaste and toothbrush were sold together in most situations, it could be a good starting point for designing new promotion!