Niche Modeling

27
Apr
In Praise of Numeracy
Posted by David Stockwell in All

Mathematical shapes can affect our lives and the decisions we make.

The
hockey stick graph describing the earths average temperature over the last millennia has been the subject of a controversial debate over reliability of methods of statistical analysis.

From this to this …

The long tail is another new icon, described in a new book, developed in the Blogosphere, by Chris Anderson called “The Long Tail”:

Forget squeezing millions from a few megahits at the top of the charts. The future of entertainment is in the millions of niche markets at the shallow end of the bit stream. Chris Anderson explains all in a book called “The Long Tail”. Follow his continuing coverage of the subject on The Long Tail blog.

As explained in Wikipedia:

The long tail is the colloquial name for a long-known feature of statistical distributions (Zipf, Power laws, Pareto distributions and/or general Lévy distributions ). The feature is also known as “heavy tails”, “power-law tails” or “Pareto tails”. Such distributions resemble the accompanying graph.

In these distributions a low frequency or low-amplitude population that gradually “tails off” follows a high frequency or high-amplitude population. In many cases the infrequent or low-amplitude events—the long tail, represented here by the yellow portion of the graph—can cumulatively outnumber or outweigh the initial portion of the graph, such that in aggregate they comprise the majority.

A niche is a concept that at minimum is the tendency of a species to prefer a particular set or range of values. To express this with a function requires a ‘hump’ or ‘inverted U’ shape centered on values optimal to the species growth and reproduction. Below are three ways to do this is with functions: step function, a truncated quadratic, and exponential.

Other examples of mathematical shapes in action can be found in the climate science journal GRL Naturally Trendy and Long term persistence in climate and the detection problem. Long term persistence or LTP refers to the tendency for the effect of random fluctuations to persist over very long time scales, giving the appearance of trends.

Readers of this blog will be familiar with the many articles written about LTP, red noise, and random series with LTP inspired by the extensive work of Koutsoyiannis in hydrology. Niche markets are important for small businesses, as they can find it profitable to serve markets too small for mainstream businesses. Anderson argues that products with low sales volume can collectively exceed the relatively few current bestsellers.

Statistical analyses collectively called ‘niche modelling’ can be applied in many areas of life. Currently I am actively developing a business delivering the analytics of niche modeling to business. More can be read about this at the site NicheShape.

Tagged as: Hockey-stick, Long Term Persistence, Prediction
Published by David Stockwell in: All
If you like this blog please take a second from your precious time and subscribe to my rss feed!

Levi Waldron

Darn, not sure how to display R code in the comments... will try again with mediawiki markup...

brownian

admin

Yeah well, longer than what? fatter than what? Thing is Chris Anderson's definition is a business with a large number of unique products, that emerges from low storage/distribution costs, compared with a business with few products. It doesn't necessarily meet the Wikipedia definition of having a preponderence of bulk (or value) residing in the tails.

In the case of fat tails it is not enough to be infinitely long and approach zero. An exponential tail like the normal curve does that and is not problematic. Its more like distributions that decay slower than exponential that are the problem and are a feature of the strangly trendy series, and processes with costly extreme events that are more frequent than predicted by normal distributions.

I think I will stop doing these teaser posts.

TCO

I think fat tail is the most intuitive, suggestive term. In either case the distribution (in some sense) is inifinitely long and approaches zero as an assmptote. Fat tail suggests that its not going down as fast as it "should".

Fat tails killed LTCM. Well and not understanding that their portfolio parts were not really independant to certain risks.

admin

Thanks for the comment Demetris. It really helps one gain an intution about the concepts to describe it that way. The behavior as a necessary result of entropy makes a lot of sense to me -- instead of seeing these behaviours as disembodied statistical distributions, that are an inevitable consequence of the 2nd law of thermodynamics. Or, from the point of view of information, a natural or complex system evolves in such a way as to minimize compressibility of its information. That is, a description of a natural system as a kind of machine with a distinct cycle or spatial scale (like a Markov process) should be the least accurate.

Demetris Koutsoyiannis

Very interesting. I like the term "long tail" better than the more common in hydrology "heavy tail" and "fat tail" or the more technical "Pareto tail" and "power-law tail", so I will adopt it from now on. I think "long tail" is more natural and expressive than the other names. For example, one says that a cat has a long tail (not a heavy tail). In addition, typologically it is closer to "long-term persistence" and "long-range dependence" (that is, long-tail autocorrelation).

Long tails in probability distribution and in autocorrelation function are two different things, both implying some scaling (power law) behaviour on the tails of these functions. In my couple of entropy papers in Hydrological Sciences Journal (2005) I used the names "state scaling" and "time scaling" for these two scaling types (the second could also be "space scaling" in a spatial process – if one substitutes space for time). Simultaneously, I tried to demonstrate in these that both long tails may be consequences of a single thing, the principle of maximum entropy, viz. maximum uncertainty.

The combined effect of long tails in probability and long tails in autocorrelation, that is the dominance of maximum entropy, makes nature fascinating, creating rich forms such as high mountain peaks, low hills and hursts, deep valleys, wide plains and intricate coastlines. Can you imagine landscapes resembling the monotony of series produced by processes such as white noise or even Markovian? Now substitute time for space (or add time to space) and you move from topography to the evolution of natural processes. The combined long tails produce time series with strange shapes such as hockey sticks, peaks and plains, with rapid and also gradual changes, or phenomena such as persistent droughts (Joseph effect), severe floods (Noah effect), movement of continents, earthquakes, etc. Destructive and catastrophic? Yes, no doubt. But it depends on the point of view. On a narrow time scale and a local, perhaps egotistic, view these phenomena are destructive and catastrophic. On a broader time scale and a more global view the same phenomena become constructive and creative. These and others even more violent have formed our cosmos.

Sign up/in

Authors

Abouts

Blogroll

Literature

In Praise of Numeracy

archives

tag cloud

Copyright © 2008 - 2010 Niche Modeling

Britney's Online Dating Websites