Data Crunchers Now the Cool Kids on Campus

Parent Cafe
21

<br>

<br>

<p>Untrue.</p>

<p>One of my son’s undergraduate database courses covered materialized views (I was a bit surprised). Now it didn’t go into a lot of depth on data warehouses but there was some coverage. I just haven’t seen any undergraduate courses that did in-depth coverage of data warehouse.</p>

<br>

<br>

<p>Access has nothing to do with how the data is stored.</p>

22

<p>texaspg - No, we do everything - retail, telecom, government, finance, even casino, any business with a lot of data. But the technology did grow out of financial.</p>

23

<p>^^Good for your son. What can you tell us about the topic beyond a college course your son has taken or point out what you don’t like about something someone else has posted? It does not seem very constructive to me.</p>

<p>At one time Walmart had the biggest data warehouse on the planet but I believe google has overtaken them lately. They were dumping everything they had into the warehouse on every single purchase despite not having the right tools to understand the patterns. The goal was to send people selective offers using information of the customers purchasing habits or notice patterns such as someone stopping buying diapers or formula and moved on to school supplies etc. Essentially, the data is being mined to identify patterns and improve sales. </p>

<p>Analytic tools are slowly catching to make this dream a reality. Essentially, one is taking the stored data and trying to predict the future.</p>

<p>oldfort - since you seem to be in the industry, can you provide a better a view of what happens than my outdated simplistic view?</p>

24

<br>

<br>

<p>You stated that those majoring in statistics need to take “data wearhousing” in order to make money. I was merely pointing out that this was untrue.</p>

<br>

<br>

<p>Well, that’s the general idea. Collect it and save it. You may not be using it now but it might be handy in the future.</p>

<br>

<br>

<p>A big part of the problem is finding people that can make use of the analytic tools. One of my co-workers worked on developing BI tools for many years and then got frustrated and went to Microsoft to work on theirs. They’ve done a much better job at it than we have. He told me that finding people to work on this stuff (both engineering the software and knowing how to use it) is difficult.</p>

25

<p>

There are tools to analyze those large data now. All I can say is that they are doing more than promotion (merchandising). They use the data to do theft detection, store expansion. They also sell their data to other business. The big thing now is to link a business’ internal data with external data to get even better info. The simplest example is to link sales data with the weather data. All very interesting.</p>

26

<p>My daughter was an English major in college and she is now a data miner at Google. Most people don’t study it in college.</p>

27

<p>[Stanford</a> CS345A: Data Mining](<a href=“http://www.stanford.edu/class/cs345a/]Stanford”>Stanford CS345A: Data Mining)
[Stanford</a> CS246: Mining Massive Data Sets (Winter 2013)](<a href=“http://www.stanford.edu/class/cs246/]Stanford”>CS246 | Home)
[STATS202</a> Data Mining and Analysis | Stanford University Online](<a href=“http://scpd.stanford.edu/search/publicCourseSearchDetails.do?method=load&courseId=1144184]STATS202”>Data Mining and Analysis Course I Stanford Online)
[Stanford</a> CS341: Project in Mining Massive Data Sets (Spring 2012)](<a href=“http://www.stanford.edu/class/cs341/]Stanford”>CS341 | Home)</p>

28

<p>^ and some offer a full program, one that I am familiar with, for example, [Institute</a> for Advanced Analytics | Dr. Michael Rappa · Information for Applicants](<a href=“Information for Applicants – Master of Science in Analytics | Institute for Advanced Analytics”>Information for Applicants – Master of Science in Analytics | Institute for Advanced Analytics)</p>

29

<p>My daughter is happy that she managed to “sneak into” data mining with an English degree. She graduated three years ago. There may be more people majoring in it since then.</p>

30

<p>Data Mining has been around for quite a while - at least since the 1990s. I think that that the interesting stores back then were OLAP (and MOLAP and ROLAP). Some practitioners may remember this paper by Jim Gray back in the 1990s:</p>

<p><a href=“http://paul.rutgers.edu/~aminabdu/cs541/cube_op.pdf[/url]”>http://paul.rutgers.edu/~aminabdu/cs541/cube_op.pdf&lt;/a&gt;&lt;/p&gt;

31

<p>I’m not saying it wasn’t around. I’m saying there probably weren’t a lot of people majoring in it and graduating in 2009. If there were my English major daughter probably wouldn’t have been hired to do data crunching. She lucked out.</p>

32

<p>The paper is very old. I think a lot has changed since then. Big data analysis in real time is a new frontier. The days of taking months or years to get a data warehouse set up to do data analysis is gone.</p>

33

<p>I agree. Jim Gray is no longer with us either. But I was just using it to show that data mining has been around for a while and that you don’t necessarily need a data warehouse to do data mining.</p>

34

<p>In-memory databases are replacing data warehouses. Faster, simpler.</p>

35

<br>

<br>

<p>The AMD64 architecture was pretty interesting back in 2003 in that it made large memory spaces available to cheap boxes and commodity software.</p>

<p>I don’t see why storage type, though, necessarily means that data warehouses are going away. Couldn’t you just put your data warehouse on an in-memory database or SSD storage? Also, what do you do if your system crashes? Do you have to rebuild your data warehouse from scratch or do you do full backup every once in a while and journal the changes so that you can roll forward from a backup? Either way implies some downtime. Maybe redundancy? Keep two copies in different locations?</p>

36

<p>With in-memory processing power, can use large columnar data table, star schema not needed.</p>

37

<p>Predictive analytics…forecasting and the common type used today.</p>

<p>prescriptive analytics…optimization…forerecasts while also suggesting future optimized actions to benefit from the predictions. The most sophisticated type of analytic often using stochastic optimization.</p>

38

<p>gman - which in memory databases are popular?</p>

39

<p>I attended a presentation on column-store by Zdonik several years ago. I think that he was working with Vertica with Stonebraker and Held back then. They later got sold to HPQ and I lost track of some of the people working there.</p>

40

<p>SAP HANA and Oracle Exalytics are the ones I’m familiar with.</p>