Tail Ratio in Explorer

TrendNeutral · September 20, 2016, 8:17am

@BobSvan2 is it possible to have a code that would allow to calculate the Tail Ratio for a bunch of systems?

https://www.quora.com/What-is-a-tail-ratio-in-a-finance-or-statistical-context

So:
download daily return %
calculate tail ratio = 95 pecentile / 5 percentile

Thanks!

BobSvan2 · September 21, 2016, 3:02pm

Hello!

Perhaps this code:


/* 
  Disclaimer: 
  The systems used in this example were selected randomly.
  Data shown in this example can contain errors.
*/

Int64[] systems = new Int64[] {
  46106678, // Bob Dylan
  90325773, // Ascendatnt
  84690231 // Genefish Hong Kong
};

// Create a TimeSheet object. Define desired data.
ITimeSheet timeSheet = TimeSheetFactory(systems, TimeInterval.Day, EquityType.Rets);

// Let TimeSheet run
timeSheet.EquitiesSheet();

foreach (var systemId in systems)
{
  Series<DateTime, Double> returnsColumn = timeSheet.GetColumnAsDouble(systemId, EquityType.Rets);
  double percentile5 = Statistics.Percentile(returnsColumn.Values, 5);
  double percentile95 = Statistics.Percentile(returnsColumn.Values, 95);
  if (percentile5 == 0.0)
  {
    TEXT = systemId.ToString() + ": TailRatio does not exist.";
  }
  else
  {
    double tailRatio = percentile95 / Math.Abs(percentile5);
    TEXT = String.Format(
                  "System {0}:   p5 = {1:N4}   p95 = {2:N4}   ratio = {3:N2}", 
                  systemId, percentile5, percentile95, tailRatio);
  }
}

H3 = "Daily returns visualized";
TABLE = GetEquitiesSheet(systems, TimeInterval.Day, EquityType.Rets);

Enjoy!

DavidStephens · September 21, 2016, 9:58pm

The above code gives wrong answers if you run multiple systems at once. It uses a timeSheet merged for all systems and that makes inaccurate daily returns percentiles for systems that haven’t traded the whole time period.

If you run the code as-is with 3 systems it calculates the following:
System 46106678: p5 = -1.0250 p95 = 1.6450 ratio = 1.60
System 90325773: p5 = -1.3825 p95 = 1.9025 ratio = 1.38
System 84690231: p5 = -0.9100 p95 = 1.1275 ratio = 1.24

Running the systems one at a time calculates different values for the two systems that haven’t traded as long as the first:
System 46106678: p5 = -1.0250 p95 = 1.6450 ratio = 1.60
System 90325773: p5 = -6.5015 p95 = 6.5970 ratio = 1.01
System 84690231: p5 = -2.4960 p95 = 2.5560 ratio = 1.02

BobSvan2 · September 21, 2016, 10:33pm

Thanks.

Please change

    timeSheet.EquitiesSheet();

to

    timeSheet.EquitiesSheet(fillLeadingMissing : false);

This eliminates “non common” bars from the beginning.

I suppose all systems are live in C2Explorer.
So they should have common bars at the end of the time sheet.
If not, we need other solution.

TrendNeutral · September 22, 2016, 12:58am

Thanks it seems to work. For the calculation of the pecentiles do we use for each system on from when the system goes life? Otherwise we would compute a lot of 0 prior to system go live…

BobSvan2 · September 22, 2016, 1:53pm

OK. Let’s do it as good as we can.

The reason for TimeSheet usage is, that it calculates returns data for every day and fills eventual gaps.
TimeSheet is used separately (one system in TimeSheet only) for each system in the following code:

// Your systems selection.
// For example - last week most popular systems.
var popularLastWeek = from stat in C2STATS
  where stat.StatName == "popular10080"
  orderby stat.StatValueVal descending
  select stat.SystemId;

// Get 10 most popular systems Ids
Int64[] systems = popularLastWeek.Take(10).ToArray();

// A list for results.
List<object> result = new List<object>();

// And something for fun.
IColumnChart chart = new ColumnChart("Tail Ratio - last week most popular systems", "Systems", "Tail Ratio");

foreach (var systemId in systems)
{
  // Get a system for Name, Started date, ...
  IC2TradingSystem system = GetC2SYSTEM(systemId);

  // Create a TimeSheet object. It calculates daily returns.
  ITimeSheet timeSheet = TimeSheetFactory(systemId, TimeInterval.Day, EquityType.Rets);

  // Let TimeSheet run for this system. 
  timeSheet.EquitiesSheet();

  // Get the Returns column for statistics.
  Series<DateTime, Double> returnsColumn = timeSheet.GetColumnAsDouble(systemId, EquityType.Rets);

  // Percentiles
  double percentile5 = Statistics.Percentile(returnsColumn.Values, 5);
  double percentile95 = Statistics.Percentile(returnsColumn.Values, 95);

  double tailRatio = 0.0;
  if (percentile5 != 0.0)
  {
    // Calculate Tail Ratio
    tailRatio = percentile95 / Math.Abs(percentile5);

    // Prepare data for output
    object trdata = new
    {
      Name = system.Name,
      Id = system.Id,
      Started = system.Started,
      Trades = system.Trades.Count(),
      Percentile5 = Math.Round((decimal)percentile5, 4),
      Percentile95 = Math.Round((decimal)percentile95, 4),
      TailRatio = Math.Round((decimal)tailRatio, 2)
    };

    // Add to the results list
    result.Add(trdata);
    // add to the chart
    chart.Add(system.Name, Math.Round((decimal)tailRatio, 2));
  }
}

CHART = chart;

HR();

H3 = "Tail Ratio - last week most popular systems";
TABLE = result;

// H3 = "Daily returns visualized";
// TABLE = GetEquitiesSheet(systems, TimeInterval.Day, EquityType.Rets);

TrendNeutral · September 23, 2016, 11:11am

Great work! would it be too much to ask to add to the column tables other key stats such as:

number of days
% win
avg win
avg loss
Sharpe
APD

TrendNeutral · September 24, 2016, 3:43am

@BobSvan2 just making sure you can see the above post. It would also be great to select all live systems with at least 180 days history. Basically we should just put the Tail Ratio in the grid

BobSvan2 · September 26, 2016, 2:25pm

Hello!

All fields from the C2ExplorerDB.c2ex_publicsystems database table are accessible from the IC2TradingSystem interface now.

A documentation is not yet updated - a list of fields is here:
See https://collective2.com/c2explorer_help/html/T_C2ExplorerDB_c2ex_publicsystems.htm

It means you can modify the Tail Ratio code like this (for example):

    object trdata = new
    {
      Name = system.Name,
      Id = system.Id,
      Started = system.Started,
      Trades = system.Trades.Count(),
      Percentile5 = Math.Round((decimal)percentile5, 4),
      Percentile95 = Math.Round((decimal)percentile95, 4),
      TailRatio = Math.Round((decimal)tailRatio, 2),
      
      // --------------- New fields -------------
      AvgWin = system.AvgWin,
      AvgLoss = system.AvgLoss,
      WinPercent = (decimal)Math.Round((double)system.NumWins / (double)system.NumTrades , 4) * 100
    };

Other info (Sharpe,…) is not so easy and need to be added from the database.

Here is an example how to get Sharpe:

TABLE = from item in C2STATS 
        where systems.Contains(item.SystemId)
        where item.StatName == "jSharpe"
        select item;

Based on this example, you can add the following code:

 // Get Sharpe
    var sharpe = (from item in C2STATS 
        where item.SystemId == system.Id
        where item.StatName == "jSharpe"
        select item.StatValueVal).FirstOrDefault();

and use the result in object trdata = new { ... }

Here is a new code with those modifications:

// Your systems selection.
// For example - last week most popular systems.
var popularLastWeek = from stat in C2STATS
  where stat.StatName == "popular10080"
  orderby stat.StatValueVal descending
  select stat.SystemId;

// Get 10 most popular systems Ids
Int64[] systems = popularLastWeek.Take(10).ToArray();

// A list for results.
List<object> result = new List<object>();

// And something for fun.
IColumnChart chart = new ColumnChart("Tail Ratio - last week most popular systems", "Systems", "Tail Ratio");

foreach (var systemId in systems)
{
  // Get a system for Name, Started date, ...
  IC2TradingSystem system = GetC2SYSTEM(systemId);

  // Create a TimeSheet object. It calculates daily returns.
  ITimeSheet timeSheet = TimeSheetFactory(systemId, TimeInterval.Day, EquityType.Rets);

  // Let TimeSheet run for this system. 
  timeSheet.EquitiesSheet();

  // Get the Returns column for statistics.
  Series<DateTime, Double> returnsColumn = timeSheet.GetColumnAsDouble(systemId, EquityType.Rets);

  // Percentiles
  double percentile5 = Statistics.Percentile(returnsColumn.Values, 5);
  double percentile95 = Statistics.Percentile(returnsColumn.Values, 95);

  double tailRatio = 0.0;
  if (percentile5 != 0.0)
  {
    // Calculate Tail Ratio
    tailRatio = percentile95 / Math.Abs(percentile5);

    // Get Sharpe
    var sharpe = (from item in C2STATS 
        where item.SystemId == system.Id
        where item.StatName == "jSharpe"
        select item.StatValueVal).FirstOrDefault();
    
    // Prepare data for output
    object trdata = new
    {
      Name = system.Name,
      Id = system.Id,
      Started = system.Started,
      Trades = system.Trades.Count(),
      Percentile5 = Math.Round((decimal)percentile5, 4),
      Percentile95 = Math.Round((decimal)percentile95, 4),
      TailRatio = Math.Round((decimal)tailRatio, 2),
      AvgWin = system.AvgWin,
      AvgLoss = system.AvgLoss,
      WinPercent = (decimal)Math.Round((double)system.NumWins / (double)system.NumTrades , 4) * 100,
      Sharpe = (decimal)sharpe
    };

    // Add to the results list
    result.Add(trdata);
    // add to the chart
    chart.Add(system.Name, Math.Round((decimal)tailRatio, 2));
  }
}

CHART = chart;

HR();

H3 = "Tail Ratio - last week most popular systems";
TABLE = result;

// H3 = "Daily returns visualized";
// TABLE = GetEquitiesSheet(systems, TimeInterval.Day, EquityType.Rets);

BobSvan2 · September 26, 2016, 2:29pm

C2Explorer includes all live systems.

TrendNeutral · September 26, 2016, 2:44pm

@BobSvan2 thanks so much! what if we wanted to compute all live systems, not just the most popular?

BobSvan2 · September 26, 2016, 2:58pm

Too much data.
There are 25746 systems there right now.
You will end with timeout.

seph · January 17, 2018, 10:23am

What is the StatName for
b (slope, estimate of beta)
and
a (intercept, estimate of alpha)
calculated with daily values, full history?

TrendNeutral · February 9, 2018, 9:12am

Hi Bob, could you tell me why the Tail Ratio code doesnt work on this system: 81877382 ?

/*
Disclaimer:
The systems used in this example were selected randomly.
Data shown in this example can contain errors.
*/

Int64[] systems = new Int64[] {
81877382

};

// Create a TimeSheet object. Define desired data.
ITimeSheet timeSheet = TimeSheetFactory(systems, TimeInterval.Day, EquityType.Rets);

// Let TimeSheet run
timeSheet.EquitiesSheet(fillLeadingMissing : false);

foreach (var systemId in systems)
{
Series<DateTime, Double> returnsColumn = timeSheet.GetColumnAsDouble(systemId, EquityType.Rets);
double percentile5 = Statistics.Percentile(returnsColumn.Values, 5);
double percentile95 = Statistics.Percentile(returnsColumn.Values, 95);
if (percentile5 == 0.0)
{
TEXT = systemId.ToString() + “: TailRatio does not exist.”;
}
else
{
double tailRatio = percentile95 / Math.Abs(percentile5);
TEXT = String.Format(
“System {0}: p5 = {1:N4} p95 = {2:N4} ratio = {3:N2}”,
systemId, percentile5, percentile95, tailRatio);
}
}

H3 = “Daily returns visualized”;
TABLE = GetEquitiesSheet(systems, TimeInterval.Day, EquityType.Rets);

TrendNeutral · March 23, 2018, 4:50pm

@BobSvan2
@MatthewKlein

Hi guys have you considered adding SPC (statistical process control) code to Explorer?

Pls look at this paper:
http://www.smallake.kr/wp-content/uploads/2016/06/Trading-Model-Uncertainty.pdf

SPC has predictive value on future returns. Below I manually used the average 5day returns for system XLN.

Excel example for XLN

It would be great to have a code that can map these control charts based on daily % returns…

BobSvan2 · March 24, 2018, 2:11pm

It works now.

Thanks.

TrendNeutral · March 29, 2018, 12:18am

@BobSvan2 @MatthewKlein
Hi guys another useful way of using the daily returns from tail ratio explorer is to apply hypothesis testing on past daily returns and test the hypothesis of whether future returns will be below or above 0. Statistics with T test can be used for this. The formualae are pretty simple , it would be great to implement this test in Explorer for a bunch of systems… See my excel attached!

T test example

BobSvan2 · March 30, 2018, 10:51am

Hello TrendNeutral!

Well, it looks like a not easy research task. I think we could add it to C2Explorer only if there is a .NET implementation with appropriate license available there.

TrendNeutral · March 31, 2018, 2:40am

Hi @BobSvan2

Agreed the SPC part is tricky. I googled the below open source codes. Could you have a look to see if you can use them?

https://statistical-process-control-spc.soft112.com/

Regarding hypothesis testing to falsify Null Hypotheses H0: average daily returns <x I believe we have all capabilities already. We just need to take the daily returns from the Tail Ratio code, get rid of the days with return = 0 (likely we are off the market) and then calculate:

n (count)
Mean
Standard deviation
t = (Mean - x) / (St Dev /sqrt(N))
p-value = T.DIST( t, n-1, 1 )
If p-value <=5% then H0 is falsified at 95% confidence and average daily returns are >0

T.Dist is an excel function but there are equivalent you can implement such as Math.Net

https://numerics.mathdotnet.com/api/MathNet.Numerics/ExcelFunctions.htm#TIn

TrendNeutral · April 2, 2018, 1:49am

@BobSvan2

I am using the code belove to generate the t test. I can’t generate the p value but the higher the t the better.

var ds = new DescriptiveStatistics(returnsColumn.Values);

double media = Math.Round(ds.Mean,2);
double numero = ds.Count;
double square = Math.Sqrt(numero);
double deviazione = Math.Round(ds.StandardDeviation,2);

double ttest = (media)/(deviazione/square);

Sorting the 100 most popular systems last week by top t value with get the following list. I copy only those with t value >2.5 as lower than that the null hypothese would be falsified for sure…

Name	Ttest
R Option	3.51
Carma Stocks	2.93
VIXTrader	2.92
VolatilityTrader	2.92
CommodityFutures	2.91
IGL CTA	2.89
Quant Models Volatility	2.88
The Momentum of Now	2.84
Optimized Partners I	2.8
QuantAlphaMR	2.77
The Vegan Growth Port	2.75
YZ income fund	2.74
Smart Volatility IRA	2.72
Long Driver Portfolio	2.7
Optimized Partners II	2.69
ETF Timer	2.67
VIXTrader Professional	2.63
Quantec VXS	2.59
NQ El Dorado	2.58
Smart Bull Portfolio	2.57
Bijagual	2.56
Tech Savvy	2.5

Topic		Replies	Views
Is there anyway to calculate the daily percentage return of a strategy? C2 Explorer	5	4669	February 18, 2017
Calculations of statistics C2 Software Development	5	710	July 15, 2019
Sharpe ratio xls? C2	2	206	December 22, 2005
Sharpe Ratio understanding C2	44	636	April 9, 2006
The grid C2	20	227	September 23, 2009

Tail Ratio in Explorer

Related topics