Category Archives: mathematica

Date functions in Mathematica

The date functions in Mathematica are in some regards excellent — you can use almost any common date format and the program will decipher it, and it computes date additions and date differences to high precision, and the date graphing functions work well. However these features come at a cost, the date functions in Mathematica are quite slow.

When writing functions that involve dates, the most important speed optimization is typically to move all date functions outside the inner loops, even if this means doing elaborate computations of all possible date combinations in advance, so that the inner loops can do table lookups for needed results. I have even created my own date functions that, though interpreted rather than compiled as Mathematica’s built in functions are, run ten times faster. (One difference being that my function is less flexible in the format of the inputs it accepts, and can proceed directly with date addition or subtraction, whereas Mathematica’s built-in function must spend epic amounts of time trying to figure out what precisely is meant by, for example “11/5/1980”. However even when you convert all Mathematica dates to AbsoluteTime[] format, it’s still slow.)

I ran some benchmarks tonight, comparing Mathematica to Visual Basic for Applications. This seems a fair comparison, as both are interpreted lanaguages. The inner loop in Mathematica looks like


{DateDifference["11/15/1997", "12/1/1998", "Day"],
DatePlus["11/15/1997", {36, "Day"}],
AbsoluteTime["December 30, 1997"]}

While the equivalent code in VBA is

diff = DateDiff("d", #11/15/97#, #12/1/98#)
sum = DateAdd("d", 36, #11/15/97#)
intTime = DateValue("December 30 1997")

Mathematica 8.01 runs that triad of lines 121 times/second, while VBA completes it about 10,000 times/second. VBA is about 82 times as fast.

An amusing quine in Mathematica

I had the idea for this Quine even before writing the longer, more traditional quine I posted yesterday. That one actually took hours to craft; this one practically wrote itself. It is based on the TextRecognize[] command that was introduced in Mathematica 8. Essentially, I wrote a line of code that prints a picture, then runs OCR on that picture and prints the result. The seed picture is an image of the program’s own source code.

Mathematica reduces the size of the picture so it fits neatly on a line of text, which makes it hard for a human to read. As they appear in the notebook, the Quine and its output look like
image quine

If we zoom in on the guinePNG image, we see
quinePic

Ta-da!

The only time consuming part of the process was selecting a font that led to an accurate OCR result. The TextRecognize command seems to do best with dictionary words, and does a comparatively lousy job of recognizing Mathematica code, which is full of odd terms and punctuation. I changed the name of the variable from “quinePNG” to “guinePNG” because TextRecognize[] was identifying the “q” as a “g” anyway.

Quine in Mathematica

A “quine” is a curious object in computer science of far-reaching implications. It is a program that generates itself as output. This is more complex than it sounds, as if you have your program source, to print that source requires that the program have something like Print[program source], but then to capture that leading print command, you need to amend your code to something like Print[“Print[”,program source,“]”], and to capture the new complexity you need another level of nesting, ad infinitum. Some people, discovering this problem upon their initial efforts to write a quine, conclude the problem is hopeless.

However, it is not at all hopeless. These can be created in every Turing-complete computer language, and in fact can be created an infinite number of ways in each language. I learned about quines recently from their Wikipedia page, and from a very nice discussion at the web page of David Madore. Reading Madore’s page, I immediately realized an extremely slick implementation that would be possible in Mathematica 8, but before trying to implement that I thought I would try a more general approach first.

Three different techniques seemed promising. The first involved putting a program that uses ToExpression[] to render a string into a string, and then run ToExpression on it. My efforts in this direction all hit infinite recursion loops.

The second direction involved using StringReplace to print a string that contained its own StringReplace command. The substring being replaced seemed to get Held, however, and I couldn’t get it to behave as desire.

The third way uses the Print command, thusly:


quine[x_String] := Print[x, ";", FromCharacterCode[10] , "quine[", InputForm[x], "]"];
quine["quine[x_String]:=Print[x,\";\",FromCharacterCode[10],\"\quine[\",InputForm[x]],\"]\""]

The output is precisely as we want —
quine

I think this meets the definition of Quine. I’ll set to the elegant Mathematica 8 only version shortly.

Another old computer — the Powerbook 180c

In 1993, Apple released the PowerBook 180c, which sold for over $4100 new. It was, I think, the final PowerBook with the iconic form factor that had been introduced with the PowerBook 100 in 1991.

powerbook

Somewhere along the way I picked one up for a percentage point or so of the original selling price. The memory had been upgraded to 14Mb, an amount that would have been astounding at the time, and almost unimaginably expensive, considering the Sumitomo Chemical Plant fire of July 1993 that caused RAM prices to spike. Consider this in comparison to the recent Tōhoku earthquake and tsunami, which put much of northern Japan at least briefly underwater and had virtually no impact on the global economy. In 1993, a fire in one Japanese factory caused computer memory prices to remain elevated for two years.

about this mac 1993

I hauled the computer out of the basement recently to convert some old floppy discs, and put it through some paces online. Unlike the Apple Lisa I recently connected, indirectly, to the web, this machine has a true TCP/IP stack, and can run mail clients, web browsers, and the like.

The machine was made before Apple routinely put ethernet ports on their computers, so an adapter is needed. There probably exist AppleTalk-to-Ethernet adapters that can be connected to the machine’s serial port. I am lucky enough to have an Asante Mini EN/SC 10T adapter, which allows older Macs to be connected to ethernet via their SCSI ports. The driver software was well coded, and the Mac regards the resulting connection as built-in ethernet.

asante en/sc 10T

Network access in Mac OS7 was handled via the MacTCP control panel, not the network control panel, which back then related only to the AppleTalk protocol I think. If you have a similar device and are having trouble configuring it, select “Ethernet Built-In” in MacTCP, then click the “More” button. If you plan to talk to the world, find the names and addresses of your DNS servers and enter them by hand. Your ISP can tell you, if you don’t use OpenDNS or another alternative.

Set your router address (called a “gateway address” here).

I do not think MacTCP handled DHCP correctly. Anyway, I set a manual IP address just in case.

macTCP 1993

Numerous older web browsers and FTP clients, etc., are available online. I used NCSA Mosaic and Fetch. Then I logged into IRC using the venerable Ircle client, which unfortunately killed itself after 30 minutes to punish me for not buying a license.
irc on a PowerBook 180c

Mathematica 2.2 was loaded on the machine, and I ran a couple of quick problems through it.
mathematica 2.2, 1993

The syntax for simple problems like this has not changed, and those problems can be solved using the exact same commands in Mathematica 8.01. On my home Mac Pro, the first problem can be solved a bit better than 8x as quickly on the modern machine, and the second problem about 127x as quickly.

Using NCSA Mosaic, here’s how this website would have looked in 1993:

monkeywrench as rendered in NCSA Mosaic

A silly article from Reuters

Reuters ran an article on the 19th titled “Wall Street’s ‘Buy Everything’ Sentiment Continues” which took as its premise that stock prices in the U.S. are rocketing upwards at an unprecedented pace, and that this, combined with low trading volumes and high levels on the VIX index, means that stocks are “due for a correction”. They quote only one person in the article, Paul Mendelsohn, sage of Charlotte, Vermont. He makes the specific claim that he has “never seen a market like this,” though he has been a “market watcher” for 35 years.

There may be reasonable explanations. Perhaps stocks were too low, so even rising rapidly, they may still be cheap. Or perhaps corporate profitability has increased significantly, so that stocks must rise rapidly to remain fairly valued. Mendelsohn does not consider such issues, saying “I’m showing, by every technical and quantitative standard I have, this market is at extreme levels. But no matter where we start out in the morning, buyers come in.”

Sadly, he doesn’t tell us what “technical and quantitative” standards he has, but the article does give us some clues of what might be worrying Mr. Mendelsohn. It claims that “Trading volume has been exceptionally low recently and the CBOE Volatility Index .VIX, Wall Street’s so-called fear gauge, is up on the week despite the gains in stocks. The index is usually inversely correlated to the S&P 500, and a rise in the VIX typically means a drop in the stock market.”

Hmm.. that sounds wrong.

normal volume

That goes back as far as Bloomberg has the data, and recent volumes don’t look unusually low to me at all. It looks pretty normal, and that volume data is from the New York Stock Exchange, which has been losing share to competing organizations like Liquidnet for years, so total recent volume is certainly higher.

How about the supposedly torrid pace of stock price increases? Reuters worries that “Wall Street posted its third consecutive week of gains with the S&P 500 now up 6.8 percent for the year and more than 20 percent in just six months.”

How weird is this? Let’s consider the 35 years of Mr. Mendelsohn’s expertise.

pretty normal
pretty normal too

It feels hard to panic over that; the current situation looks pretty much like any other bull market. Volume is normal, the pace of increase has plenty of precedent. Through most of the previous periods that looked like this, you’d have been very happy to continue to hold the market.

Oh, about the last claim, that an increase in VIX predicts a drop in the stock market? I have heard this from others, and rigorously testing it is complex (what is the proper lag between an increase in VIX and a drop in the market? Do we care about the absolute level of the VIX, or relative changes therein? etc.) However, as a quick check I just ran normalized VIX against normalized SPX (total return), since January 1990, since that’s when Bloomberg’s data for the VIX starts.

"negatively correlated?"

There’s no evident pattern there, but even the most mindless chart junky willing to trust a regression line however inappropriate has to accept that the Reuters has this one wrong — the best fit is a positive correlation.

All the graphs above come from Mathematica, using the Mathematica link to Bloomberg. The graphs with the superimposed pink rectangles were a little challenging to make; here’s the source code if you want to do something similar —

rangesAndScale =
Select[Table[{{spxDailyLong[[2]][[i]][[1]],
spxDailyLong[[2]][[i + 33]][[
1]]}, (spxDailyLong[[2]][[i + 33]]/spxDailyLong[[2]][[i]])[[
2]] - 1}, {i, 1,
Length[spxDailyLong[[2]]] - 34}], #[[2]] > .0679 &];


ranges = Transpose[rangesAndScale][[1]];

(* we want overlapping rectangles to be unified into a smaller number of wider rectangles, so we need the following *)

unifier[rangeList_] := Module[{i, outp}, outp = {};
For[i = 1, i <= Length[rangeList], i = i + 1, If[i < Length[rangeList], If[AbsoluteTime[rangeList[[i + 1]][[1]]] <= AbsoluteTime[rangeList[[i]][[2]]], outp = Append[ outp, {rangeList[[i]][[1]], rangeList[[i + 1]][[2]]}]; i = i + 1, outp = Append[outp, rangeList[[i]]] ], outp = Append[outp, rangeList[[i]]]] ]; outp ]

unifiedRanges = FixedPoint[unifier[#] &, ranges] ;


legendRects =
Map[{Pink, Opacity[.5],
Rectangle[{AbsoluteTime[#[[1]]], -100}, {AbsoluteTime[#[[2]]],
1600}]} &, unifiedRanges];

DateListPlot[spxDailyLong[[2]], Frame -> {True, True, False, False},
Epilog -> legendRects,
PlotLabel ->
"\"I've never seen a market like this\"\npink=markets just like \
this (6.78% rise in 34 trading days)"]