Inside the Calculation

Rootclaim takes a deep, data-driven look at the issues that interest society. The platform integrates all available evidence, assesses it for credibility and uses probabilistic models to reach conclusions about the likelihood of competing hypotheses.

Rootclaim is grounded in two pillars:

1. Proven probabilistic inference models – The model breaks down highly complex issues into small questions that are each answerable by humans, and then uses these answers to reach mathematically indisputable conclusions.
2. Openly crowdsourced evidence and claims –Anyone can impact an analysis by contributing evidence, rational explanations, past examples and statistics. Unlike polling or voting, a strong claim by one person can beat many widely supported weaker claims.

Rootclaim outperforms human reasoning by correcting for the biases and flaws of human intuition. Its conclusions represent the best available understanding of the complexity and uncertainty in our world.

Instead of first looking at all the evidence, and then trying to figure out which hypothesis is the best match, start from the hypotheses: Supposing Hypothesis A were true, how well does each piece of evidence fit with that storyline? This reversal is the key to Rootclaim’s approach. It lets us break down impossibly complex issues into a series of answerable questions.

Second, all the inputs, from every side of the issue, are combined and sent through Rootclaim’s Bayesian probabilistic engine. Unlike other types of analyses, Rootclaim doesn’t cherry-pick the evidence. Since anyone can add new information or challenge the existing inputs, all the evidence, from every side of the issue, is balanced together using Rootclaim’s Bayesian inference algorithm, letting us reconstruct the complex issue at hand.

Finally, the engine outputs its conclusions, showing which hypothesis came out on top. The conclusions are a direct mathematical result of the inputs, so if the inputs are right, the outputs are indisputable.

Remember, the whole analysis is transparent and open to scrutiny. If the conclusions don't seem right to you, challenge the inputs and improve the results!

Each analysis begins with a question, and compares several possible hypotheses (answers). For example, let’s say we want to know the answer to the following question:

Once the question and hypotheses are established, all the information related to the question is collected and assessed for its reliability. There are two main types of information evaluated in the analysis:

1. Starting point: The analysis begins with data based on past cases and general statistics, in order to find out the baseline likelihood for each hypothesis. In this case, we would need to find out the overall incidence of lung cancer and chronic bronchitis in the population. (These statistics are for this example only, and are not necessarily accurate.)
   Starting Point for Hypothesis 1: Incidence of lung cancer = 0.05%
   Starting Point for Hypothesis 2: Incidence of chronic bronchitis = 5%
2. Evidence: Beyond the general statistics gathered for the starting point, the analysis collects all the specific information related to the case at hand, and evaluates how reliable the source is for each piece of evidence. For this example, let’s say we have two piece of evidence:
   Specific Evidence A: Mr. X has shortness of breath. (Source reliability = 99%)
   Specific Evidence B: Mr. X has a chronic cough. (Source reliability = 95%)

Finally, the relationship between each hypothesis and each piece of specific evidence is evaluated: How likely is each piece of evidence, given each hypothesis? In this example, there would be four different hypothesis-to-evidence relationships to assess:

1. If Mr. X has lung cancer, how likely is it that he would have shortness of breath?
2. If Mr. X has lung cancer, how likely is it that he would have a chronic cough?
3. If Mr. X has chronic bronchitis, how likely is it that he would have shortness of breath?
4. If Mr. X has chronic bronchitis, how likely is it that he would have a chronic cough?

Each new piece of evidence added to the analysis must be assessed in the same way, asking how likely that evidence would be under each of the competing hypotheses. The numbers that represent each of these relationships must be supported by solid reasoning. Anyone in the crowd can challenge these inputs and propose a different number, backed by better reasoning.

Ready to start contributing to a story? Each part of the analysis is discussed in further detail below.

Each competing hypothesis is one possible answer to the main question. All of the hypotheses must be mutually exclusive (two can’t be true at the same time), and collectively exhaustive (there are no other reasonable answers to consider). As a group, the competing hypotheses capture all the main points of view surrounding the topic under debate.

Mutually exclusive and collectively exhaustive

In this example, the hypothesis choices are exhaustive because one of the hypotheses must be true, and they are mutually exclusive because if one hypothesis is true then the other(s) must be false.

Each hypothesis should meet these criteria:

1. It is not highly implausible (e.g. “Santa Claus did it”).
2. A better hypothesis is not available that is similar in essence. For example, if the public mostly cares whether the culprit is A or B, we won't investigate every variation—just the most likely version of events for each possibility.
3. It cannot have been clearly debunked by straightforward evidence (no complex probability calculations required).

In some cases, it is impossible for the list of hypotheses to truly be collectively exhaustive, since there may be an infinite number of possible answers to the question at hand. Even in these situations, the list of hypotheses is still assumed to be collectively exhaustive for the sake of mathematical validity.

The evidence includes all the relevant information known about the topic at hand, compiled and organized by category. Each piece of evidence is supported by a source, whose reliability is assessed independently.

The Essence

It’s important to phrase the evidence in a way that conveys the essence of the claim. In other words, it should contain exactly the amount of information that is relevant to the analysis--no more, and no less.

Depending on what the relevance to the analysis is, the essence of this evidence might look like this:

In a different analysis, however, the essence might look like this:

As this example demonstrates, the essence of the evidence depends on the context of the analysis.

Starting point

The starting point of each analysis consists of general statistics and past examples related to each hypothesis. This general information is used to calculate how plausible each hypothesis is without regard to specifics of the case at hand.

Why is this necessary? If a hypothesis starts out already unlikely (e.g., “Santa did it!”), one would need a lot more evidence to prove that it’s true. If a hypothesis is already reasonable, one would need less evidence to back it up.

The similar cases considered must be specific enough to reflect the essential differences between hypotheses, but general enough to create a large enough sample to infer meaningful statistics.

Source reliability

Each piece of evidence is supported by a source, which is treated as an additional node in the Bayesian network. The source’s reliability is measured as one minus the probability that the source would provide such a report, if the report weren’t in fact true.

For example, if a source’s reliability score is 95%, then that means there's a 5% probability that this source would report such evidence when that evidence is false. When the information was brought to the public through several sources, we evaluate the weakest link in the chain of communication.

Conflicts among sources and evidence

Sometimes, the way the evidence is phrased has to account for dependencies among sources, evidence, and hypotheses, in order to allow for all the hypothesis-to-evidence relationships to be assessed on equal footing.

If, for example, the source of a piece of evidence is a potential suspect in the crime under investigation, then the reliability of that suspect as a source might change dramatically depending on whether we’re looking at the hypothesis in which the suspect is guilty versus the hypothesis in which he is innocent.

This discrepancy is resolved by changing the definition of the evidence to the report of the event, rather than the event itself.

Instead of this wording:

The evidence must be phrased like this:

Now the hypothesis-to-evidence relationship can account for the evidence directly, without any potential differences in the reliability of the source.

How to assess source reliability

To assess the source reliability, we try to estimate the probability that the source will provide such a report if it weren’t true. The first step in this process is estimating the most likely reason that the source would make a false report.

While it is possible that some sources might knowingly lie, there are other reasons that a source might report something other than the truth – like flaws in the raw data, or innocent misinterpretation of the data.

The source reliability questionnaire opens with exactly this question:

Depending on the answer to that question, a reliability score is assigned to the source. This score applies only to this specific piece of evidence - it is not a general score of the source’s overall reliability related to other topics, since a source may be more reliable in certain subjects than in others.

The full source reliability questionnaire is reproduced here:

1. Sensory fallibility: unfavorable neutral conditions.
   What is the likelihood that the source observed/obtained/remembered the raw information the claim is based on accurately with respect to physical or other neutral conditions?
   1. Extremely unlikely (99.999%)
   2. Unlikely (90%)
   3. Somewhat unlikely (60%)
   4. Somewhat likely (40%)
   5. Almost certain (1%)
2. Analytical fallibility: lack of technical capability or domain expertise.
   What is the likelihood that the source is capable (technically, expertise-wise) of making the claim based on the raw information?
   1. Extremely likely (99.999%)
   2. Likely (90%)
   3. Somewhat likely (60%)
   4. Somewhat unlikely (40%)
   5. Almost certain (1%)
3. Subconscious bias: implicit prejudice.
   What is the likelihood that the source misinterpreted information used to make the claim due to the source’s own inherent bias but not in a manner that can be viewed as being untruthful?
   1. Almost impossible (99.999%)
   2. Unlikely (90%)
   3. Somewhat unlikely (60%)
   4. Somewhat likely (40%)
   5. Almost certain (1%)
4. Deliberate misreporting of information.
   Taking into account Benefit and Cost, what is the net benefit for the source to alter or misreport the information the claim is based on?
   1. Huge net benefit
   2. Major net benefit
   3. Moderate net benefit
   4. Minor net benefit
   5. Almost no net benefit
   Taking into account Risk and Penalty, what is the expected cost for the source to alter or misreport the information the claim is based on?
   1. Huge expected cost
   2. Major expected cost
   3. Moderate expected cost
   4. Minor expected cost
   5. Almost no expected cost

1. Yes
   If yes: Enter the numeric confidence value asserted by the source (i.e. the p-value or confidence interval).
2. No
   If no: What language does the quote use to assert the level of confidence in the claim?
   1. 100% - For example: Is (no qualifiers), true, without question, assuredly, certainly, incontestable
   2. 99% - For example: With few exceptions, almost certainly, virtually always
   3. 95% - For example: Highly likely, very likely, very probably
   4. 90% - For example: Countless, in general, on the whole, likely, probably
   5. 80% - For example: Seems to, apparently, appears to, somewhat likely, arguably, evidently, reportedly, seemingly, somewhat, often, very, in many cases
   6. 75% - For example: Suggests
   7. 50% - For example: Inconclusive, could, possibly, maybe, may, might, perhaps, plausibly, conceivably, hypothetically
   8. 40% - For example: May suggest, in some cases, in my opinion
   9. 30% - For example: Allegedly
   10. 25% - For example: Somewhat unlikely
   11. 10% - For example: Unlikely, doubtful, improbable, few
   12. 5% - For example: Highly unlikely, very unlikely, very improbably
   13. 1% - For example: Rarely, hardly any, highly unlikely, almost never
   14. 0% - For example: False, never, impossible

In general, nothing is 100% certain. However, in some specific cases, the certainty is so high that for the purposes of simplicity and readability of the analysis, we set its likelihood at 100% :

1. We can usually assume that the report of what happened (i.e. “The New York Times reported that the event happened”) is true, unless there is a conflict between sources and evidence .
2. If the piece of evidence at hand is clearly shown in a photo or video, we can usually assume that what is shown is, in fact, what happened (unless there is a specific, credible reason to believe the photo or video misrepresents reality in some way).
3. We can also assume a piece of evidence is true if it is the logical conclusion of a series of claims, each of which is nearly certain.

At any time, a piece of evidence set as 100% certain may be contested and/or analyzed in further depth. A piece of evidence (or source) is only considered 100% unless and until it is challenged by the crowd.

Hypothesis-to-Evidence Relationship

Instead of asking how likely each hypothesis is given the evidence, the inputs to the probabilistic engine reverse the direction of the analysis: how well does each piece of evidence fit under each of the competing hypotheses?

For example, let’s say the competing hypotheses are:

And the evidence is:

Bayesian network structure

What is a dependency?

Two pieces of evidence are dependent if a change in the likelihood of the first produces a change in the likelihood of the second. For example, take the following two pieces of evidence:

If Bob got more sleep, would this affect how well he would do on his math test? If the answer is yes, then there is a dependency between these two pieces of evidence.

What happens if two pieces of evidence are dependent?

Formally, Bayesian networks assume that all of the dependencies among nodes in the graph are explicitly represented by arcs (connections) between them. In other words, all nodes of the network should be independent given their parent (the node pointing to them). In order to make the analysis mathematically valid, therefore, we have to address this dependency by changing the structure of the Bayesian network:

In order to make the analysis mathematically valid, therefore, we have to address this dependency by changing the structure of the Bayesian network:

Sub-analyses

A sub-analysis pits all the competing possible causes for the dependency against one another. For example, there could be a couple of different reasons that could explain why Bob failing his math test and only sleeping for four hours are dependent pieces of evidence:

All pieces of evidence whose dependencies are resolved by the addition of the sub-analysis are positioned in the network as child nodes of the sub-analysis node. Evidence and sources are evaluated exactly the same way as in the main analysis, but instead of looking at the likelihood of the evidence under each hypothesis, we look at the likelihood of the evidence under each scenario.

At the level of the main analysis, each of the competing scenarios in the sub-analysis are evaluated exactly the same way that the hypothesis-to-evidence relationship evaluates other evidence. However, just like with the hypotheses, the scenarios in a sub-analysis must be mutually exclusive and collectively exhaustive, given the hypotheses above. This means that the likelihoods of the scenarios given each hypothesis must sum to 100%.

By structuring these competing possible causes as a sub-analysis, the dependency is resolved.

Storyline assumptions

Storyline assumptions are another possible tool that can be used to resolve dependencies among multiple pieces of evidence. If there is a reasonable common cause that can explain all the dependent pieces of evidence, and there are no other reasonable common causes competing with it, then that common cause is integrated as an assumption directly into the storyline of the hypothesis itself.

How does this work?

Taking into consideration this third piece of evidence (where he explicitly states that he didn’t study), it seems very implausible that Bob studied for the math test.

This leaves only one reasonable common cause that would explain all three of these pieces of evidence. So instead of creating a sub-analysis, we can incorporate the most reasonable explanation into the storyline of the hypothesis itself:

Each storyline assumption added makes the hypothesis more specific, and therefore it also makes the hypothesis less likely. In order to determine how much less likely this new, more specific version of the hypothesis is, we evaluate the storyline assumption similar to the hypothesis-to-evidence relationship: supposing the hypothesis were true, what is the likelihood that this new storyline assumption would also be true?

The rest of the analysis should also be adjusted so that the new storyline assumption is considered part of the hypothesis. This can affect the evaluation of the hypothesis-to-evidence relationships, as well as the way evidence is phrased, and even how source reliability is assessed.

By adding the most likely storyline assumption, and adjusting the rest of the network based on the more specific version of the hypothesis, the dependency is resolved and the adjusted hypothesis is the most likely it can be. Its initial likelihood is lower (since it is more specific) but the conditional probability of the evidence given this hypothesis is higher.

How storyline assumptions are incorporated into the calculation

The prior probability (starting point) of each hypothesis is multiplied by the likelihood assigned to each storyline assumption, given the hypothesis. Then, these results are normalized to yield the adjusted prior probability for the hypothesis.

For example, let’s say the starting point for an analysis gives Hypothesis 1 a prior probability of 90% and Hypothesis 2 a prior probability of 10%. Hypothesis 1 requires one storyline assumption, which was assessed as 40% likely given the original hypothesis. Hypothesis 2 requires no storyline assumptions.

The calculation looks like this:

For the purposes of the Bayesian inference algorithm, the normalized adjusted prior probability (the last line in the table above) is treated as the prior probability in the Bayesian network.