Back of the Envelope

Observations on the Theory and Empirics of Mathematical Finance

Archive for the ‘Computing’ Category

[WP] Shiny for Finance

leave a comment »

For those familiar with R, Shiny offers a great alternative to spreadsheets. Targeted at instructors, we have a new paper introducing Shiny for finance applications in the classroom. Here is a longish abstract:

Despite the popularity of open-source languages like R and Python in modern empirical research and the data-science industry, spreadsheet programs like Microsoft Excel remain the data analysis software of choice in much of the business-school curriculum, including at IIMA. Even if instructors are comfortable with modern programming languages, they have to pitch their courses at the level of computer literacy prevalent among students. Excel then appears to be a natural choice given its popularity, but this choice constrains the depth of analysis that is possible and requires a certain amount of dumbing-down of the subject by the instructor. Recent software advances however make the ubiquitous web browser a worthy challenger to the spreadsheet. This article introduces one such browser-based tool called Shiny for bringing finance applications to the classroom and smart phones. Fueled by the availability of high-quality R packages in finance and statistics, Shiny brings together the power of HTML with the R programming language. It naturally creates an environment for the instructor to focus on the role of parameters and assumptions in analysis without the clutter of data, and allows the instructor to go beyond the toy problems that are necessitated by the nature of spreadsheets. The learning curve is short for an interested instructor with even a rudimentary exposure to programming in any language. The article ends with the discussion of a fully-worked out example of Shiny for teaching the mean variance efficient frontier in a basic investments course.

The working paper is here. It needs a bit of fine tuning yet, but comments welcome.

Written by Vineet

April 4, 2017 at 12:30 pm

[OS] Moving to Xubuntu

with 2 comments

About a year and a half back I talked about my transition to Lubuntu. I found it pretty perfect for my usage, and have had no major issues with it but Chrome has been misbehaving a bit since I moved to LTS 16.04. While fixes exist, I couldn’t find anything permanent, and since I had some time I thought it might be a good opportunity to try the slightly more endowed Xubuntu – with a slightly larger user base while remaining sufficiently lean.

In the last couple of days, I have tested Xubuntu on my office desktop (i5) and my Asus laptop (i3). Its boot time is similar to Lubuntu, and Chrome and Firefox open quickly enough (if anything, a bit quicker than Lubuntu). Many settings are similar to those in Lubuntu (e.g. setting proxy), but setting keyboard shortcuts is even easier. So far I am not missing Lubuntu.

Building upon the earlier discussion on Lubuntu, here are my settings for a minimalist Xubuntu (on an i3 machine with 4 GB RAM). I use them in the following order:

  1. Disk partitioning scheme:
    /boot = 4 GB, /root = 16 GB, /swap = 4 GB, /tmp = 4 GB
    /opt = 6 GB, /var = 6 GB, /usr = 24 GB, /usr/local = 16 GB

    /home takes up the remaining space. I also leave about 24 GB space free for emergency.

  2.  First update:
    sudo apt-get update && sudo apt-get upgrade && sudo apt-get install aptitude
  3. Essentials:
    sudo aptitude install synaptic xubuntu-restricted-extras chromium-browser
    sudo aptitude install gdebi libboost-all-dev swig cmake git xdotool vlc
    sudo aptitude install python-gpgme mencoder curl libcurl4-openssl-dev
    sudo aptitude install usb-creator-gtk p7zip keepass2 cryptsetup cryptmount
    sudo aptitude install lame alsaplayer-common alsaplayer-text soundconverter
    sudo aptitude install pdftk pdfchain djview4 zathura zathura-ps zathura-djvu
    sudo aptitude install gparted gimp getmail4 gnome-search-tool ibritish
  4. At this stage I set-up Dropbox in a separate partition and let the folders sync. In the meantime I create the necessary documentation infrastructure:
    sudo aptitude install emacs auctex pandoc markdown texlive-full

    You may install auctex and ess directly from the emacs package manager (containing the more recent versions). I install ess after getting R.

  5. Keyboard shortcuts: Settings Manager -> Settings Editor -> xfce4-keyboard-shortcuts
  6. Printer: If you have a duplex printer, and it does not automatically respect duplex settings, you may have to set it to ON from Settings -> Printer. If you have an HP printer, hplip-gui (available from synaptic) is also useful.
  7. Speedup boot: 
    • Remove unnecessary start-up services (and add any) from Settings Manager -> Session and Startup
    • Create a file ~/.gtkrc-2.0 and paste the following: Hat-tip BinaryTides
      gtk-menu-popup-delay = 0
      gtk-menu-popdown-delay = 0
      gtk-menu-bar-popup-delay = 0
      gtk-enable-animations = 0
      gtk-timeout-expand = 0
      gtk-timeout-initial = 0
      gtk-timeout-repeat = 0
  8. Python-specific: Hat-tip  Chris’ sandbox
    sudo aptitude install gfortran g++ build-essential pkg-config
    sudo aptitude install python-dev python3-dev python-pip python3-pip
    sudo aptitude install python3-tk libatlas-dev liblapack-dev libblas-dev
    sudo aptitude install libfreetype6 libfreetype6-dev libpng-dev libjpeg8-dev

    I no longer use Anaconda for Python and install all necessary packages within virtual environments (a useful intro is here (option 3)). Jorgen Schaefer’s elpy package means that I don’t have to leave emacs at all.

    In particular, after setting the virtual environments, I have the following installed packages (in that order) for data science/financial modeling related work in emacs:

    pip install numpy scipy pandas matplotlib
    pip install sympy seaborn jupyter scikit-learn statsmodels
    pip install rope jedi flake8 importmagic autopep8 yapf

Written by Vineet

January 9, 2017 at 8:53 pm

[Etc] Finance without spreadsheets

leave a comment »

As part of the syndicated “View from IIMA” series, my article appeared in livemint yesterday . (I just wish the sub-editor had taken a bit more effort to write the blurb than just snipping a sentence off the text, with the first person singular and all!)

Here is the unabridged version with hyperlinks.

Finance without spreadsheets

What is common to Barclays, Fidelity and JP Morgan, other than the fact that they are all large multinational financial institutions? As the regular readers of this daily would probably recognize, all the three have had to face the embarrassment of acknowledging spreadsheet errors amounting to hundreds of millions of dollars.

It has not only been the financial services firms that have been left red-faced due to “Excel mistakes”. MI5, the British intelligence agency ended up bugging hundreds of wrong phones in 2011 due to formatting errors, and the 2012 London Olympics organizing committee oversold 10000 tickets because of wrong ‘data entry’. Such examples abound, just Google ‘spreadsheet errors‘. And by all accounts it seems that people have learned to live with it, and, thanks to regulators, some have even found ways to make money off it.

The Basel Committee on Banking Supervision explicitly instructs banks to have “effective mitigants in place” against errors creeping in due to “manual processes”. The industry has naturally responded, and many risk consulting firms today provide advisory services to manage spreadsheet risk. Not many outside the risk management world would probably be aware, but there even exists a European Spreadsheet Risks Interest Group.

If one looks back at the evolution of spreadsheet software since the days of Lotus 1-2-3, the look-and-feel apart, nothing much has changed in substance. Yes, a user has more tricks up her sleeve and there are web apps to work in groups now, but, fundamentally, the data in Microsoft Office Excel is still organized and analysed in a tabular form.

The first version of Excel dates back to 1985, and what is astonishing is that for almost 30 years much of the corporate world has been using versions of the same software sold by one firm. As a comparison, despite the market share enjoyed by Maruti Suzuki Ltd, its cars today are no longer the obvious choice for most upper-middle class Indians, and one would have thought loyalty would matter more for automobiles where after-sales service network is valued. But in the corporate world, the larger the organization, more prevalent seems to be the use of Excel.

Now lest this piece be seen as a rant against Excel or spreadsheets, let me admit that I am no anti-spreadsheet evangelist. I used to be an Excel user in my earlier job, and continue to use LibreOffice for quick, back of the envelope calculations. There is no denying the utility of a spreadsheet software, and given the ubiquity of Microsoft systems in organizations, perhaps even unavoidable. It remains useful for prototyping, basic data analysis and teaching elementary statistics, but, in my opinion, it is not exactly suited for large projects and serious scientific research.

As Harvard professors Carmen Reinhart and Kenneth Rogoff found out, despite the quality and substance of their research, a couple of avoidable spreadsheet errors unnecessarily diverted attention from the main message of their work. And then there is the famous London Whale incident. As Mint’s readers would remember, it took more than a year for JP Morgan to get that monkey off its back, and not without its CEO Jamie Dimon having to take a huge salary cut.

I can’t speak for other areas, but use of spreadsheets for quantitative finance – by which I mean portfolio analysis, financial engineering and risk management – is just inefficient, if not outright lazy. Even at the basic level, given the quantum and nature of data involved, it just invites too many bad practices – cell references, manual inputs and copy-and-paste habits, virtually unreadable formulas, unwieldy formatting, multiple tabs and resultant humongous file sizes, over-use of colors with clumsy legends, misleading and ugly graphs, and the list goes on. Yes, I know many of these can be obviated, and existing power users are smart about it, but am not sure beginners appreciate the pitfalls until it’s too late.

More than reflecting problems with any particular spreadsheet software, what these examples illustrate is a lack of context of the scale and scope involved in the choice of computing environment. Working in large projects, with interconnected data and modern techniques, is best done in an environment which facilitates efficient separation of raw data, its organization, analysis and output. A spreadsheet software is simply isn’t up for it.

Perhaps Newton’s first law of motion has something to do with it too, but this inertia in the industry has meant that many business schools continue to predominantly use spreadsheets for quantitative courses. While jobs in the data science and analytics world has meant more acceptance of the programming languages R and Python in ‘elective’ courses, much of the basic curriculum still relies on spreadsheets.

All this, of course, goes back to the more fundamental issue of whether a business school curriculum should be proactive or reactive. As far as the choice of computing environment in curricula is concerned, most business schools have been in the reactive mode, likely because of placement pressures. The students, of course, do not have an incentive to shift and prefer the status quo, as most naturally want to ‘blend in’. The question is whether these arguments hold water in today’s environment of easy availability of MOOCs on all kinds of things quantitative, including free training courses in Excel, R and Python. Not in my opinion.

When it comes to doing complex quantitative analysis and using cutting-edge methods, spreadsheets simply lack the statistical sophistication of R and the power of Python. In fact, both today come with their web apps – Shiny for R and Jupyter for Python – that makes illustrating and sharing applications a cinch, and fun.

An unfortunate response to lack of ability of spreadsheets to do modern statistical analyses in finance has been to purchase proprietary/black-box software, many of which often simply adapt and package the existing freely available open-source programs. This not only results in an inefficient use of a business school’s money, but also leads to perpetuation of bad habits, as for commercial reasons most software try to give their software the look-and-feel of spreadsheets. It is also problematic from a pedagogical standpoint as one is implicitly telling students to rely on someone else’s implementation without worrying about the details. The more courses use such software, stronger this reinforcement to rely on black-box software. To say that ‘my teacher used that’ or ‘US schools do that’ is, at best, naive.

A common refrain against switching to R/Python is that their use requires programming and ‘I am not from engineering/science background’. It is like saying that I do not want to learn English because I studied in a school where the medium of instruction was vernacular. Yes, as with any new language, no denying there is a set-up cost involved, but not more than about a week from an interested user.

Although we still have a long way to go, I believe that among Indian business schools, IIMA is at the forefront of introducing open-source computing environment to its students, both at our post-graduate and doctoral programs. We have done away with our compulsory Excel course in the first year, and many of us use R and Python in the classroom regularly. A few of us in the finance area have not only completely shifted to open-source environment for our research and teaching but have also completely ditched the proprietary operating systems in favor of Ubuntu, or even Arch Linux. Now that’s a topic for another day.

Written by Vineet

March 15, 2016 at 1:01 pm

Posted in Computing, Stories

Tagged with ,

[CompFin] Installing QuantLib and QuantLib-Python in Windows

with 13 comments

In our recent paper introducing QuantLib, we had an appendix which described how to set both QuantLib and QuantLib-Python up for use in Ubuntu and Windows. Since I rarely use Windows these days, for instructions for Windows we mostly relied on what was available on the QuantLib website and answers to relevant questions on stackoverflow.

When I got my hands on a Windows laptop recently, to my embarrassment I realized I could not use those instructions to get QuantLib and QuantLib-Python working on a Windows 7 machine. Needless to say this needed sorting out, and now it has been.

I’ll have the paper updated on our website shortly, but in the mean time here are the instructions for installing QuantLib and QuantLib-Python in Windows. These have been tested on three different Windows machines, so we are reasonably sure they work. However, if something doesn’t work for you, do drop in a comment.

Step-by-step by instructions for installing QuantLib and QuantLib-Python in Windows

  1. QuantLib requires a working C++ Environment, and for the purposes of instructions here (meant for a novice), it is recommended that the user install the free Microsoft Visual Express Desktop 2013 edition which comes with Visual C++ 12 (MSVC12).
  2. QuantLib-Python requires a working Python environment, and we recommend working with the Anaconda meta-package. Instructions are available from the Anaconda install page. For our purpose here, Windows users should work with the 32-bit version of Anaconda. It is recommended to install Anaconda with all the default settings, and update it by running \texttt{conda update conda} and \texttt{conda update anaconda}. Again, it does not matter if one installs Anaconda-2.x (with Python 2.7) or Anaconda-3.x (with Python 3), but it is recommended that the user install the Python 3 version, as that’s where the Python language seems to be headed.
  3. Compiling QuantLib requires Boost C++ libraries installed. And this is where the utility of working with MSVC becomes most apparent. Pre-packaged binaries for Boost are available for specific versions of MSVC (in our case MSVC12) from its sourceforge page, and installing Boost on MSVC is no more difficult than installing any other software in Windows. (Users should download the executable for the 32-bit architecture, as the free version of MSVC does not seem to contain the necessary utilities for compiling in 64-bit. There is no such problem if you have access to a professional edition of MSVC though.)
  4. After installing Boost, we need to install SWIG. Pre-packaged binaries for SWIG for Windows are available from its source page (latest version 3.0.5), and it is enough to extract the SWIG zip file in a convenient folder.
  5. Download the same version of QuantLib and QuantLib-Python (latest version 1.6)
  6. Launch the Visual Studio command prompt: All the commands below assume that we are working in Visual Studio command prompt (or if not, all the relevant Visual Studio related environment variables have been set). It can be launched from the Start menu (in Windows 7) or from Apps (in Windows 8). If you can’t find it, see here.
    • NB: Though we haven’t encountered any issue when using MSVC12, as pointed out by Andres in comments, when working with VS 2015, it might be safest to work with x64 Native Tools Command Prompt.
  7. After the above steps, installing QuantLib in Windows simply requires the following steps (mind the gaps):
     cd \path\to\QuantLib-1.6
    set myboost=\path\to\boost\boost_1_58_0
    msbuild /p:AdditionalLibPaths="%myboost\lib32-msvc-12.0" /p:Configuration=Release /p:Platform=Win32 QuantLib_vc12.sln 

    The last step above may take a while (on an Intel i5, Windows 7 machine with 4 GB RAM it took almost 45 minutes).

    Note that throughout in this post \texttt{\textbackslash path\textbackslash to\textbackslash someplace} represents the directory of \texttt{someplace}. For example, if \texttt{boost\_1\_58\_0} is installed in \texttt{C:\textbackslash boost}\texttt{\textbackslash path\textbackslash to\textbackslash boost\_1\_58\_0} should be taken to mean \texttt{C:\textbackslash boost\textbackslash boost\_1\_58\_0}.

  8. One should ensure that QuantLib is correctly installed in the previous step, as what follows assume a working QuantLib. If it is correctly installed, you should see something like the following message at some stage before the installation completes:
     Tests completed in 16m 50s
    Test suite "Master Test Suite" passed with:
    1002021 assertions out of 1002021 assertions passed
    609 test cases out of 609 passed 

    And at the very end:

     Build succeeded.
    0 Warning(s)
    0 Error(s)
    Time Elapsed 00:44:20.07
  9. Although not absolutely necessary, to ensure that all is well, it is recommended to also build QuantLib in Debug mode (replace ‘Release’ by ‘Debug’ in the \texttt{msbuild} command in step 7 above; and will take roughly half the time it took to build the Release version), and test the code in the QuantLib Usage section on the QuantLib install page for Windows.
  10. Installing QuantLib-Python requires a few extra settings. At the Visual Studio command prompt, set the following (again, mind the gaps):
     cd \path\to\QuantLib-SWIG-1.6\Python
    set PATH=\path\to\Anaconda3;\path\to\Anaconda3\scripts;\path\to\swigwin-3.0.5;\path\to\QuantLib-1.6\lib;%PATH%
    set QL_DIR=\path\to\QuantLib-1.6
    set INCLUDE=\path\to\boost_1_58_0;%INCLUDE%
    set LIB=\path\to\boost\lib32-msvc-12.0;%LIB%
    echo [build] > setup.cfg && echo compiler=msvc >> setup.cfg 
  11. Once the environment variables have been set in step 10 above, all we need is to run the following steps:
     cd \path\to\QuantLib-SWIG-1.6\Python
    python wrap
    python build
    python test
    python install 

    If you are wondering how would you know that QuantLib-Python has been installed correctly, after the \texttt{python test} step, you should see a message something like this:

    Ran 36 tests in 1.253s
  12. To be doubly sure, you should now go to Luigi Ballabio’s QuantLib page and run the examples there.

My co-author has a more detailed post on his blog discussing this, and in addition to providing motivation for some of our choices, he also describes how to install QuantLib and QuantLib-Python using only open-source tools (for “free as in free speech”) and without using Visual Studio. Highly recommended.

Update (8th July): Paper updated with these instructions.

Written by Vineet

July 7, 2015 at 2:16 am

[OS] Lubuntu: A minimalist operating system for open-source applications

with one comment

Both Windows and Ubuntu today come bundled with many applications which are of no use to most users. Separating wheat from the chaff, however, is not all that easy, or extremely tedious.

Fortunately, for those of us in research/academia who do not rely on proprietary software, and use their computers mainly for reading (using a pdf-reader), internet (using Chrome/Firefox), writing (using LaTeX, and sometimes needing basic word-processing and spreadsheet-ing), programming (using Python/R), and rely on open-source applications for these, there are two wonderful alternatives out there: Lubuntu and Xubuntu.

Both are extremely lightweight and easily customizable. You may choose either of them, and basic instructions for installing these are available on their homepages. (If you are a novice and in doubt, download the the LTS version 14.04).

FWIW, below I provide instructions for a very basic set-up of Lubuntu (because that’s what I use) for a user who mainly uses Emacs, LaTeX, Python, R and Chrome.

Before you go ahead and start following these, let me get the usual disclaimer out of the way. While the instructions below have worked for me and I happily continue to use Lubuntu, please use them at your own risk. The standard no-warranty disclaimer as in GNU GPL v3 applies.

1. Disk-partitioning scheme: While installing, Lubuntu gives user the choice to either install everything in a single partition, or ‘do something else’ and custom-partition the disk. For those who don’t understand what this means, ignore this step and move to step 2 below. Else, read on.

There are many resources out there describing optimal disk-partitioning schemes depending on the size of the hard-disk/RAM and whether or not you have a dual-boot system, but in my experience, for both standalone and dual-boot installations, the following works quite fine (but again, if you have more than 1 TB machine, you should be more liberal with /opt, /usr, /usr/local and /var):

  • /boot = RAM
  • /root = 4 x RAM
  • /swap = 4 x RAM
  • /tmp = min(4 x RAM, 16GB)
  • /opt = min(4 x RAM, 16 GB)
  • /usr = max(8 x RAM, 96GB)
  • /usr/local = max(4 x RAM, 64GB)
  • /var = max(2 x RAM, 24GB)
  • /home = remaining

The alternative, of course, is to simply ignore partitioning and install everything using the default option provided by Lubuntu installer.

2. Setting up Network: If you are working on a wired PC, you need to ask your network administrator for the following: Address (IP), Network, Gateway and DNS. If you are working on a laptop, you need the WIFI SSID, security-type and possibly user login and password.

If you are behind a proxy, you need to additionally tell that to Lubuntu. To do this, follow step 1 (“Set up the proxy/proxies for most programs”) and step 3 (“Finally, set up the proxies for apt-get and Update Manager”) of instructions from the accepted answer at askubuntu here.

3. First update after install:

  • sudo apt-get update && sudo apt-get upgrade
  • sudo apt-get install lubuntu-restricted extras
  • sudo apt-get install synaptic
  • Run Update manager from Applications

4. After first update and synaptic installation: Launch synaptic and select and install the following

  • emacs (metapackage), texlive-full, auctex, ess, pdftk, pdfchain, vlc, gimp, 7zip, ibritish, djview4, gnome-search-tool, python-gpgme, curl, libcurl4-openssl-dev, libreoffice

5. Other third-party software:

  • Download Chrome, R and Dropbox from their homepage and follow the instructions there. For Dropbox, make sure that for the first time all files are already copied to your desired Dropbox location, else it may take eternity to sync if your connection is slow.

6. Resolving the sound-card issue (in case you can’t hear audio in your machine):

  • Open the following file as super-user:
    • sudo leafpad /etc/modprobe.d/alsa-base.conf
  • Add the following: options snd-hda-intel index=1

7. Setting start-up applications in Lubuntu:

  • Menu (Alt-F1) -> Preferences -> Default applications for LXsession -> Autostart

8. Keyboard shortcuts: For those who shy-away from using mouse and prefer keyboard to navigate, the following link lists all the available shortcuts: If you’d like to create custom keystrokes for things you use often, look here for an example (a life-saver for me really):

9. Anaconda for Python: For installing Anaconda for Python, I quote from our recent working paper:

Although Python is available as a standalone library from the official project page, we recommend working with meta-packages like Anaconda (made available for free by the Continuum Analytics company as on date) or Canopy (made available for free by Enthought Inc. as on date). The advantage is that it is easier to install support packages and update the Python library from within these meta-packages. We have worked (and continue to work) with the Anaconda meta-package, and found it extremely convenient to work with. Instruction for installing Anaconda are available from the Anaconda install page, and we have encountered no problem in installing it in Windows or Ubuntu.In Ubuntu, if there are going to be multiple users accessing the same machine, it is advisable to install Anaconda in “\opt\” directory rather than in “\home\user\” (default) directory. Again, it does not matter if one installs Anaconda-2.x or Anaconda-3.x, but it is recommended that the user install the python3 version, as that’s where the Python language seems to be headed.

That should be enough. If something doesn’t work for you or if you have any comments, please drop a line.

Written by Vineet

April 3, 2015 at 10:00 pm