Quantopian Review and Comparison to AmiBroker

In my last post, Avoiding Trades Before Earnings, I mentioned that I used Quantopian to do the research. Several readers asked about my thoughts about Quantopian and how it compares to AmiBroker. Some asked if I had left AmiBroker for Quantopian. What follows are my impressions after using Quantopian for several months and how it compares to AmiBroker.

The big question is will I be switching from AmiBroker to Quantopian for my backtesting?

My comments are from the point of a view of a stock trader who uses daily data.

For those Quantopian experts out there, please point out any errors or omissions in the comments and I will fix the text in the blog.

I do realize that some of the issues that I bring up would be mitigated if I did not use Quantopian and used Zipline and Python directly. But that requires a lot more work for the typical user. The focus of this review is on using Quantopian as a backtesting platform

What is Quantopian?

From their site: “Quantopian provides free education, data, and tools so anyone can pursue quantitative finance. Select members license their algorithms and share in the profits.”

From Wikipedia: “The company has a two-sided market business model: The first side consists of algorithm-developer members who develop and test for free, focusing on algorithm development for factors that can be added to Quantopian’s offerings to institutional investors . All members can compete against other members in a series of contests called the “Quantopian Open.” Anyone can join the site and (optionally) enter the contests: no particular educational qualification nor work experience are required. Quantopian provides them with free data sources and tools, largely built in the Python programming language. The second side is institutional investors. Serving them will become the main focus of Quantopian. These members have their investments managed by the winning algorithms. Successful developer-members can get a royalty or commission from investor-members, who profit from the former’s algorithm used with larger resources > $1M.”

The Data

The best part about Quantopian is the data. It has lots of free data, including 1 minute and daily data for US Stocks back to 2002 with delisted stocks. For daily data, going back to 1999 is preferred for me because I like using from 2000-2006 for out-of-sample testing. But for free this is great!

They also have lots of free non-price data like Twitter and StockTwits Trader Mood and Sentdex Sentiment Analysis. Read more at Quantopian Data.

And then there is the fundamental data. This caught my attention. It has Factset Fundamentals data up to until a year ago.  Some of this data you must pay for but the prices seem reasonable. Most come with free sample data which makes testing your idea possible to see if it is worth paying for the most up to date data. I used the free data for the Avoiding Trades Before Earnings post.

Quantopian does not have current or historical constituent data for the US Indexes. I have come to depend on this data from Norgate Data. What Quantopian provides is the Q500US & Q1500US, which is the most liquid stocks as calculated by them. Read more The Q500US and Q1500US. This gives you something similar to the S&P500.

Quantopian has futures data back to 2002 in both 1 minute and daily timeframes for 72 different trading instruments. You can read more about that data here.

Trade Execution

All your trades are executed using the one-minute bars, even your daily bar trading strategy. You tell Quantopian which bar to execute your trade. This sounds nice but it comes with some issues. You can only enter on the 9:32amET bar. So, there is no entering at the open. And the closing value of the last bar of the day is not the closing value you would see in Yahoo or Norgate Data or most other data sources. Quantopian uses the last trade on any exchange. From their FAQ:

Quantopian uses the last traded price as the close price for the security. Depending on the data source, others may use end-of-day (EOD) prices. For example, Yahoo is an EOD datasource. Yahoo and other EOD data providers get their price and volume data from the official exchange record. Quantopian’s data is generated by the actual trades, regardless of what exchange the trade was made on. The EOD sources rarely exactly match data derived from intraday data. For instance, the official close for a NYSE stock is the last trade of the day for the stock on NYSE. But if the stock also trades on Chicago, Pacific or another regional exchange, the last trade on one of those exchanges could be our close.

What I don’t like about this is that you cannot check your data against any other data sources. Sometimes your data seems wrong and it always good to check with other sources.

External Data

Sometimes, I use data that is not supplied by my data provider. In this case I want to be able to easily import this data and use it. It is possible to do in Quantopian but there are lots of steps involved. You can read how to do this in Quantopian at Upload Your Custom Datasets and Signals with Self-Serve Data. In AmiBroker, it takes less than a minute to import data and to use it.

Python

Quantopian uses Python as its programming language. Overall this is a big plus if you know no programming language because there are lots of book and websites on how to program in Python. If you already know Python, then it is even better. No need to learn a custom language like AmiBroker’s AFL, which is C like.

Code Editor

The Quantopian code editor is OK. It is missing some features that I expect from a professional level editor such as bookmarks and search/replace. But it does have my main two features that I like: code folding and auto-complete.

Code Simplicity

For simple strategies, it takes about 20 lines of code in Quantopian vs 5 lines in AmiBroker. But once you get into more complex strategies, I found that number of lines of code being about the same. As to which code is easier to write and read, that is a personal preference. I find both Quantopian and AmiBroker code easy to follow.

Community

The forums for any product are important because this is where you go to learn and get your questions answered. A great feature of Quantopian forums is that people share their algorithms and code. Quantopian makes it easy with one button press for you to copy that code for your own use.

I had read that Quantopian has over 210,000 members. But how many of these are active users and then how many participate in the forums? I looked at the number of posts with a comment in the last week. I found 54 posts. Is that a lot or not? I then did the same thing for the AmiBroker forums and found 62 posts. Both boards have about the same activity which surprised me.

Indicators

Quantopian has all the usual indicators which is expected. But there is a gotcha that forum participants fell into quite often. Certain indicators like RSI and EMA require more data than the length/periods parameter you pass in because of the smoothing calculation. For example, for a 14 period RSI, you need about 30 data points before the RSI settles down near the correct value. In Quantopian you ask for amount of data first and then pass that to RSI. So, I would frequently see people get 15 bars of data and then pass those 15 bars to the 14-day RSI calculation. This would produce an RSI14 value but not the correct one.

This is annoying because this either requires one to know which indicators are like this or to always ask for more data.

Also depending on which RSI function you call you may get either the Wilder’s Smoothing or Cutler smoothing but the documentation does not make this clear. You can read more about this at RSI calculation.

Backtest Speed

Quantopian has the concept of a “Build Algorithm” and “Full Backtest.” The former does a quick test without all the reporting, while the later does everything. All comments on Quantopian backtests are using “Full Backtest.”

Given that Quantopian is cloud based, this is where I was concerned. I have become spoiled with AmiBroker’s speed. On Quantopian, a simple RSI2 strategy on the SPY takes 6 minutes to run and 1 second on AmiBroker.

A 10 year, RSI2 pullback strategy on the Q500US takes 25 minutes to run. While in AmiBroker it takes 10 seconds to run.

Read those times again! AmiBroker is over 100 times faster.

Now, I will be the first to admit, my Python code might not be the most efficient but it can’t be that bad.

Optimization

To some optimization is a dirty word associated with over-fitting. To me it is an important part of the testing process. Quantopian cannot optimize on parameters. Here is a recent discussion about it Parameter Optimization: Is it possible? on the forums.

Backtest Reports

The full backtest report, which they call full tear sheet, for a Quantopian run has lots of interesting portfolio statistics and charts. But to see this report, you must run additional code after running your backtest. For AmiBroker, the reports are automatically generated during those 10 seconds.

In AmiBroker, it is relatively easy to add your own custom backtest statistics and charts. I could not figure out if this is possible in Quantopian. If someone knows how to do this, can you provide a link in the comments below?

Positions Report gives you the daily positions you are in, the value, and gains. I like this.

Transactions Report gives you the daily buy and sells. Another nice report.

One thing that Quantopian makes easy is plotting of up to seven values to see. If you wanted to see how many positions you had open per day or percent invested, this is easy to do. This only requires a couple lines of code. This is definitely more work in AmiBroker.

Annoyingly, there are no trade stats is this report. To get those, you must run yet again more code to get trade stats. This report provides a good amount of useful trade stats. I liked that it provided average and median numbers.

Want a trade list? A row for each trade with the date entered, price entered, date exited, exit price, return of the trade and more? This is so basic that I could not even think about not getting one.

Quantopian has no trade list.

How do you then look at your trades to make sure they are right? That you are executing what you want? Or better yet, copy into Excel so you can do more analysis. Those tasks appear to be impossible when backtesting with Quantopian. I really hope some one proves me wrong and tells me how to do this.

Getting Reports/Data to Excel

One cannot copy the Positions or Transactions reports. That means you cannot get it into Excel to do additional analysis.

You can copy the data from the full tear sheet. This is portfolio statistics and charts.

You can copy the data from the trade analysis tear sheet. This is trade statistics and charts.

Cloud vs Local

All your code lives on the Quantopian servers. You must trust them to not look at it and have it secure enough that it does not leak out. Depending on how paranoid one is, this can be a big issue. I am on the fence if this alone would stop me. It is something to be aware of.

Final Thoughts

For my needs, AmiBroker is a much better backtesting platform than Quantopian.

Who should consider Quantopian?

• You don’t want to pay for daily data
• You want one-minute data. This is expensive to buy
• You know Python already
• You want to use fundamental data

Why I will continue to use AmiBroker over Quantopian

• Backtesting speed
• Ability to optimize
• Trade list
• Custom reports and charts
• Choice of data providers so that I can get the features (like index membership), trading instruments, and markets that I need

For me, it is AmiBroker for my backtesting and consulting needs.

Backtesting platform used: AmiBroker. Data provider: Norgate Data (referral link)

Good quant trading,