Split type-checking into interface and implementation in parallel workers

[ilevkivskyi]

The general idea is very straightforward: when doing type-checking, we first type-check only module top-levels and those functions/methods that define/infer externally visible variables. Then we write cache and send new interface hash back to coordinator to unblock more SCCs early. This makes parallel type-checking ~25% faster.

However, this simple idea surfaced multiple quirks and old hacks. I address some of them in this PR, but I decided to handle the rest in follow up PR(s) to limit the size of this one.

First, important implementation details:

• On each select() call, coordinator collects all responses, both interface and implementation ones, and processes them as a single batch. This simplifies reasoning and shouldn't affect performance.
• We need to write indirect dependencies to a separate cache file, since they are only known after processing function bodies. I combine them together with error messages in files called foo.meta_ex.ff. Not 100% sure about the name, couldn't find anything more meaningful.
• Overload signatures are now processed as part of the top-level in type checker. This is a big change, but it is unavoidable and it didn't cause any problems with the daemon.
• Initializers (default values of function arguments) are now processed as part of the top-levels (to match runtime semantics). Btw @hauntsaninja you optimized them away in some cases, I am not sure this is safe in presence of walrus, see e.g. testWalrus.
• local_definitions() now do not yield methods of classes nested in functions. We add such methods to both symbol table of their actual class, and to the module top-level symbol table, thus causing double-processing.

Now some smaller things I already fixed:

• We used to have three scoping systems to track current class in type checker. One existed purely for the purpose of TypeForm support. I think two is enough, so I deleted the last one.
• AwaitableGenerator return type wrapping used to happen during processing of function body, which is obviously wrong.
• Invalid function redefinitions sometimes caused duplicate errors in case of partial types/deferrals. Now they should not, as I explicitly skip them after emitting first error.
• Some generated methods were not marked as such. Now they are.

Finally, some remaining problems and how I propose to address them in followups:

• Narrowing of final global variables is not preserved in functions anymore, see testNarrowingOfFinalPersistsInFunctions. Supporting this will be tricky/expensive, it would require preserving binder state at the point of each function definition, and restoring it later. IMO this is a relatively niche edge case, and we can simply "un-support" it (there is a simple workaround, add an assert in function body). To be clear, there are no problems with a much more common use of this feature: preserving narrowing in nested functions/lambdas.
• Support for --disallow-incomplete-defs in plugins doesn't work, see testDisallowIncompleteDefsAttrsPartialAnnotations. I think this should be not hard to fix (with some dedicated cleaner support). I can do this in a follow-up PR soon.
• Around a dozen incremental tests are skipped in parallel mode because order of error messages is more unstable now (which is expected). To be clear, we still group errors per module, but order of modules is much less predictable now. If there are no objections, I am going to ignore order of modules when comparing errors in incremental tests in a follow-up PR.
• When inferred type variable variance is not ready, we fall back to covariance, see testPEP695InferVarianceNotReadyWhenNeeded. However, when processing function/method bodies in a later phase, variance is ready more often. Although this is an improvement, it creates an inconsistency between parallel mode, and regular mode. I propose to address this by making the two-phase logic default even without parallel checking, see below.
• Finally, there are few edge cases with --local-partial-types when behavior is different in parallel mode, see e.g. testLocalPartialTypesWithGlobalInitializedToNone. Again the new behavior is IMO clearly better. However, it again creates an inconsistency with non-parallel mode. I propose to address this by enabling two-phase (interface then implementation) checking whenever --local-partial-types is enabled (globally, not per-file), even without parallel checking. Since --local-partial-types will be default behavior soon (and hopefully the only behavior at some point), this will allow us to avoid discrepancies between parallel and regular checking. @JukkaL what do you think?

[ilevkivskyi]

Oh btw, @JukkaL I think there is a bug in misc/diff-cache.py that may cause spurious diffs, see a TODO I added.

[ilevkivskyi]

All things in (small) mypy_primer are either good or neutral.

[hauntsaninja]

Could be worth adding a test for the discord.py improvement

[JukkaL]

I'm planning to test this on a big internal repo (probably tomorrow). I'll also try parallel checking again -- last time memory usage was too high to use many workers, but things should be better now.

[JukkaL]

I'm seeing mypy parallel run crashes with this PR when type checking the biggest internal codebase at work, but I'm not sure if they are caused by this -- this may just change the order of processing so that a pre-existing issue gets triggered. I will continue the investigation after the long weekend.

[ilevkivskyi]

@JukkaL can you post a traceback (and maybe a snippet of code where the crash happens)? It may well be some implicit assumption breaks when type-checking functions after top-levels.

[JukkaL]

The internal codebase generates some syntax errors because of an issue with the native parser. After working around the syntax errors, the parallel run completes, so the crashes may be related to syntax errors. However, there are a handful of false positives. Also, this regresses performance -- now parallel checking with two workers is slower than sequential checking (about 10% slower), on macOS. On master parallel checking with two workers is about 13% faster (which is still not great).

When looking at top output while a parallel run is active, each worker process is at only about 80% to 90% CPU utilization. It's possible that the added communication/synchronization overhead slows things down, at least on macOS. I'll measure performance on Linux next. I will also try to reproduce the crashes and provide tracebacks.

[ilevkivskyi]

Also, this regresses performance -- now parallel checking with two workers is slower than sequential checking (about 10% slower), on macOS. On master parallel checking with two workers is about 13% faster (which is still not great).

TBH this is really weird. Can you try running with --dump-build-stats to understand why it is so slow? Even the master number is worse than what you mentioned before in the very first PR #20280 (comment), while self-check performance improved ~50% since then.

[JukkaL]

I used 2 workers above instead of 3 in my older comment. I can try using 3 workers as well, I think I should have enough RAM for it.

[ilevkivskyi]

Ideas about how to make this better:

Some of the items we are already doing, and I am not going to do any of the rest. Performance on Linux seems good (btw do you have numbers for Linux with multiple workers?) If you (or anyone else) wants to work on Mac, you can do it in your own time.

[JukkaL]

I don't have full numbers for Linux, but here are the ones I have (for split bodies only):

• one process (old parser): 698s
• one process (new parser): 640s
• 1 worker: 707s
• 3 workers: 354s

The overhead from multiple processes is much smaller compared to macOS. It's likely faster than sequential on two workers already, which sounds like a reasonable baseline performance target.

I can continue working on macOS performance afterwards (doesn't block this PR). I have both personal and work mac laptops, so I can run measurements in a relatively clean environment without extra security software. I can measure parallel self check on my personal mac tomorrow (probably).

I'm also planning to create a parallel checking synthetic benchmark with many small files, to measure coordination overhead. We can also add separate benchmarks with larger files, but it looks like the small file one would be the most helpful at this point.

[ilevkivskyi]

Couple observations:

• I added some more performance stats locally, specifically around all send()/receive() calls. On both Linux and Mac the communication overhead is only 1-2% on self-check. So the problem may actually be with something else.
• Very bad performance on self-check on Mac that I observed was actually caused by the fact that I have Python 3.14 there, while I normally use Python 3.12 on Linux. It turns out there were major changes in GC logic in Python 3.14, so that our GC freeze hack now works against us (more precisely gc.freeze() calls get ~1000x slower if there are gc.unfreeze() calls in between them). By simply disabling the GC freeze hack I see ~1.5x speed-up with 4 workers compared to sequential on Mac on current master (with no visible effect of the GC freeze hack on sequential cold runs on Python 3.14).

I can continue working on macOS performance afterwards (doesn't block this PR).

I think we should focus on landing currently open PRs first, so that all building blocks are in place. In particular, this PR introduces (inevitable) semantic differences. I think we should agree on what to do with them. I wrote a whole lot of discussion about this in the PR description. But on the other hand, it is hard to keep so many moving parts in my head, so I am going to start merging soon.

I will make another couple PRs one for more performance stats, and will try to adjust GC freeze hack so that it works on all Python versions for both sequential and parallel runs (this may be non-trivial).

[JukkaL]

I will focus on getting more information about the crash and the false positives that I mentioned above so that we can move forward with this PR soon.

Since we have ideas about addressing the macOS bottlenecks (even if we might not fully understand the problem yet), this can happen separately from this PR (and I can work on them).

[JukkaL]

Here's a simplification of one of the new false positives from the big internal codebase, which seems similar to the testLocalPartialTypesWithGlobalInitializedToNone case discussed in PR description, and seems benign:

# mypy: local-partial-types

class C:
    x = None  # type: ignore[var-annotated]

    def f(self) -> object:
        if not C.x:
            C.x = 1  # New error here with split bodies
        return C.x

[ilevkivskyi]

@JukkaL Yeah, I think this one is fine. Btw I get the same error with --allow-redefinition-new without split bodies. In some sense this split logic makes local partial types even a bit more "local".

[JukkaL]

Another simplified regression:

from typing import Any, Callable, overload

@overload
def option(*, callback: Callable[[str], object] = ...) -> Any: ...
@overload
def option(*, callback: Callable[[int], object] = ...) -> Any: ...
def option(**kwargs: object) -> None: pass

@option(callback=lambda x: [y for y in x])  # Error here
def f() -> None: pass

When using --num-workers=1, it generates this error (with --num-workers=0, there is no error):

t.py:9: error: "int" has no attribute "__iter__"; maybe "__int__"? (not iterable)  [attr-defined]

[JukkaL]

I have one more potential regression. I'll investigate it tomorrow.

[ilevkivskyi]

@JukkaL Yeah, that exposes an existing bug, here is a repro without splitting (fails on master):

[case testDeferredLamdaInOverload]
from typing import Any, Callable, overload

@overload
def option(*, callback: Callable[[str], object] = ...) -> Any: ...
@overload
def option(*, callback: Callable[[int], object] = ...) -> Any: ...
def option(**kwargs: object) -> None: pass

def test() -> None:
    option(callback=lambda x: [y for y in x])
    x = C().x

class C:
    def __init__(self) -> None:
        self.x = defer()

def defer() -> int: ...
[builtins fixtures/primitives.pyi]

Fix should be simple: reset lambda argument types when visiting in empty context. As I mentioned before this PR may uncover many weird edge cases where type checker is not idempotent.

[JukkaL]

The final potential regression actually looks like a fixed bug.

First we need m.pyi:

from typing import NewType

class C:
    V = NewType('V', int)
    X = C.V(0)

Now this file has a new error when using split bodies (--num-workers=1):

# mypy: warn-redundant-casts

from enum import IntEnum
from typing import cast

import m

def g(type: m.C.V) -> None: ...

def f(type: C) -> None:
    g(cast(m.C.V, type.value))  # new error: Redundant cast to "V"

class C(IntEnum):
    X = m.C.X

However, it looks like the type of type.value is now correct, as the cast actually seems redundant, so this looks fine. Apparently the forward reference gets an invalid type on master.

[ilevkivskyi]

@JukkaL yeah, the last one looks like an improvement. I didn't look in details, but the bug may be that the attribute hook doesn't (properly) defer the node. So that our own plugin makes a mistake similar to the one causing #18579.

[ilevkivskyi]

Actually CheckerPluginInterface doesn't even expose handle_cannot_determine_type(). This would be another (relatively) significant change, so should be its own PR IMO, I am however going to add a fix for deferred lambda in overload here.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

discord.py (https://github.com/Rapptz/discord.py)
- discord/backoff.py:63: error: Incompatible default for parameter "integral" (default has type "Literal[False]", parameter has type "Literal[True]")  [assignment]
+ discord/backoff.py:63: error: Incompatible default for parameter "integral" (default has type "Literal[False]", parameter has type "T")  [assignment]
- discord/interactions.py:1109: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]
- discord/interactions.py:1255: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]
- discord/interactions.py:1645: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]
- discord/webhook/async_.py:969: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]

cki-lib (https://gitlab.com/cki-project/cki-lib)
- cki_lib/krb_ticket_refresher.py:26: error: Call to untyped function "_close_to_expire_ticket" in typed context  [no-untyped-call]
+ cki_lib/krb_ticket_refresher.py:26: error: Call to untyped function "_close_to_expire_ticket" of "RefreshKerberosTicket" in typed context  [no-untyped-call]

freqtrade (https://github.com/freqtrade/freqtrade)
- freqtrade/templates/FreqaiExampleStrategy.py:245: error: Invalid index type "tuple[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...], list[str]]" for "_LocIndexerFrame[DataFrame]"; expected type "tuple[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...], Hashable]"  [index]
- freqtrade/templates/FreqaiExampleStrategy.py:245: error: Unsupported left operand type for & ("tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]")  [operator]
- freqtrade/templates/FreqaiExampleStrategy.py:245: error: Argument 2 to "reduce" has incompatible type "list[Series[builtins.bool]]"; expected "Iterable[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]]"  [arg-type]
- freqtrade/templates/FreqaiExampleStrategy.py:255: error: Invalid index type "tuple[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...], list[str]]" for "_LocIndexerFrame[DataFrame]"; expected type "tuple[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...], Hashable]"  [index]
- freqtrade/templates/FreqaiExampleStrategy.py:255: error: Unsupported left operand type for & ("tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]")  [operator]
- freqtrade/templates/FreqaiExampleStrategy.py:255: error: Argument 2 to "reduce" has incompatible type "list[Series[builtins.bool]]"; expected "Iterable[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]]"  [arg-type]
- freqtrade/templates/FreqaiExampleStrategy.py:263: error: Unsupported left operand type for & ("tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]")  [operator]
- freqtrade/templates/FreqaiExampleStrategy.py:263: error: Argument 2 to "reduce" has incompatible type "list[Series[builtins.bool]]"; expected "Iterable[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]]"  [arg-type]
- freqtrade/templates/FreqaiExampleStrategy.py:267: error: Unsupported left operand type for & ("tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]")  [operator]
- freqtrade/templates/FreqaiExampleStrategy.py:267: error: Argument 2 to "reduce" has incompatible type "list[Series[builtins.bool]]"; expected "Iterable[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]]"  [arg-type]

[ilevkivskyi]

The new entry in the primer (freqtrade) looks good as well.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

discord.py (https://github.com/Rapptz/discord.py)
- discord/backoff.py:63: error: Incompatible default for parameter "integral" (default has type "Literal[False]", parameter has type "Literal[True]")  [assignment]
+ discord/backoff.py:63: error: Incompatible default for parameter "integral" (default has type "Literal[False]", parameter has type "T")  [assignment]
- discord/interactions.py:1109: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]
- discord/interactions.py:1255: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]
- discord/interactions.py:1645: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]
- discord/webhook/async_.py:969: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]

cki-lib (https://gitlab.com/cki-project/cki-lib)
- cki_lib/krb_ticket_refresher.py:26: error: Call to untyped function "_close_to_expire_ticket" in typed context  [no-untyped-call]
+ cki_lib/krb_ticket_refresher.py:26: error: Call to untyped function "_close_to_expire_ticket" of "RefreshKerberosTicket" in typed context  [no-untyped-call]

freqtrade (https://github.com/freqtrade/freqtrade)
- freqtrade/templates/FreqaiExampleStrategy.py:245: error: Invalid index type "tuple[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...], list[str]]" for "_LocIndexerFrame[DataFrame]"; expected type "tuple[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...], Hashable]"  [index]
- freqtrade/templates/FreqaiExampleStrategy.py:245: error: Unsupported left operand type for & ("tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]")  [operator]
- freqtrade/templates/FreqaiExampleStrategy.py:245: error: Argument 2 to "reduce" has incompatible type "list[Series[builtins.bool]]"; expected "Iterable[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]]"  [arg-type]
- freqtrade/templates/FreqaiExampleStrategy.py:255: error: Invalid index type "tuple[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...], list[str]]" for "_LocIndexerFrame[DataFrame]"; expected type "tuple[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...], Hashable]"  [index]
- freqtrade/templates/FreqaiExampleStrategy.py:255: error: Unsupported left operand type for & ("tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]")  [operator]
- freqtrade/templates/FreqaiExampleStrategy.py:255: error: Argument 2 to "reduce" has incompatible type "list[Series[builtins.bool]]"; expected "Iterable[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]]"  [arg-type]
- freqtrade/templates/FreqaiExampleStrategy.py:263: error: Unsupported left operand type for & ("tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]")  [operator]
- freqtrade/templates/FreqaiExampleStrategy.py:263: error: Argument 2 to "reduce" has incompatible type "list[Series[builtins.bool]]"; expected "Iterable[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]]"  [arg-type]
- freqtrade/templates/FreqaiExampleStrategy.py:267: error: Unsupported left operand type for & ("tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]")  [operator]
- freqtrade/templates/FreqaiExampleStrategy.py:267: error: Argument 2 to "reduce" has incompatible type "list[Series[builtins.bool]]"; expected "Iterable[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]]"  [arg-type]

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

discord.py (https://github.com/Rapptz/discord.py)
- discord/backoff.py:63: error: Incompatible default for parameter "integral" (default has type "Literal[False]", parameter has type "Literal[True]")  [assignment]
+ discord/backoff.py:63: error: Incompatible default for parameter "integral" (default has type "Literal[False]", parameter has type "T")  [assignment]
- discord/interactions.py:1109: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]
- discord/interactions.py:1255: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]
- discord/interactions.py:1645: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]
- discord/webhook/async_.py:969: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]

cki-lib (https://gitlab.com/cki-project/cki-lib)
- cki_lib/krb_ticket_refresher.py:26: error: Call to untyped function "_close_to_expire_ticket" in typed context  [no-untyped-call]
+ cki_lib/krb_ticket_refresher.py:26: error: Call to untyped function "_close_to_expire_ticket" of "RefreshKerberosTicket" in typed context  [no-untyped-call]

freqtrade (https://github.com/freqtrade/freqtrade)
- freqtrade/templates/FreqaiExampleStrategy.py:245: error: Invalid index type "tuple[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...], list[str]]" for "_LocIndexerFrame[DataFrame]"; expected type "tuple[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...], Hashable]"  [index]
- freqtrade/templates/FreqaiExampleStrategy.py:245: error: Unsupported left operand type for & ("tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]")  [operator]
- freqtrade/templates/FreqaiExampleStrategy.py:245: error: Argument 2 to "reduce" has incompatible type "list[Series[builtins.bool]]"; expected "Iterable[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]]"  [arg-type]
- freqtrade/templates/FreqaiExampleStrategy.py:255: error: Invalid index type "tuple[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...], list[str]]" for "_LocIndexerFrame[DataFrame]"; expected type "tuple[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...], Hashable]"  [index]
- freqtrade/templates/FreqaiExampleStrategy.py:255: error: Unsupported left operand type for & ("tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]")  [operator]
- freqtrade/templates/FreqaiExampleStrategy.py:255: error: Argument 2 to "reduce" has incompatible type "list[Series[builtins.bool]]"; expected "Iterable[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]]"  [arg-type]
- freqtrade/templates/FreqaiExampleStrategy.py:263: error: Unsupported left operand type for & ("tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]")  [operator]
- freqtrade/templates/FreqaiExampleStrategy.py:263: error: Argument 2 to "reduce" has incompatible type "list[Series[builtins.bool]]"; expected "Iterable[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]]"  [arg-type]
- freqtrade/templates/FreqaiExampleStrategy.py:267: error: Unsupported left operand type for & ("tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]")  [operator]
- freqtrade/templates/FreqaiExampleStrategy.py:267: error: Argument 2 to "reduce" has incompatible type "list[Series[builtins.bool]]"; expected "Iterable[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]]"  [arg-type]

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

discord.py (https://github.com/Rapptz/discord.py)
- discord/backoff.py:63: error: Incompatible default for parameter "integral" (default has type "Literal[False]", parameter has type "Literal[True]")  [assignment]
+ discord/backoff.py:63: error: Incompatible default for parameter "integral" (default has type "Literal[False]", parameter has type "T")  [assignment]
- discord/interactions.py:1109: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]
- discord/interactions.py:1255: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]
- discord/interactions.py:1645: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]
- discord/webhook/async_.py:969: error: Incompatible default for parameter "delay" (default has type "float | None", parameter has type "float")  [assignment]

cki-lib (https://gitlab.com/cki-project/cki-lib)
- cki_lib/krb_ticket_refresher.py:26: error: Call to untyped function "_close_to_expire_ticket" in typed context  [no-untyped-call]
+ cki_lib/krb_ticket_refresher.py:26: error: Call to untyped function "_close_to_expire_ticket" of "RefreshKerberosTicket" in typed context  [no-untyped-call]

freqtrade (https://github.com/freqtrade/freqtrade)
- freqtrade/templates/FreqaiExampleStrategy.py:245: error: Invalid index type "tuple[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...], list[str]]" for "_LocIndexerFrame[DataFrame]"; expected type "tuple[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...], Hashable]"  [index]
- freqtrade/templates/FreqaiExampleStrategy.py:245: error: Unsupported left operand type for & ("tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]")  [operator]
- freqtrade/templates/FreqaiExampleStrategy.py:245: error: Argument 2 to "reduce" has incompatible type "list[Series[builtins.bool]]"; expected "Iterable[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]]"  [arg-type]
- freqtrade/templates/FreqaiExampleStrategy.py:255: error: Invalid index type "tuple[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...], list[str]]" for "_LocIndexerFrame[DataFrame]"; expected type "tuple[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...], Hashable]"  [index]
- freqtrade/templates/FreqaiExampleStrategy.py:255: error: Unsupported left operand type for & ("tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]")  [operator]
- freqtrade/templates/FreqaiExampleStrategy.py:255: error: Argument 2 to "reduce" has incompatible type "list[Series[builtins.bool]]"; expected "Iterable[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]]"  [arg-type]
- freqtrade/templates/FreqaiExampleStrategy.py:263: error: Unsupported left operand type for & ("tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]")  [operator]
- freqtrade/templates/FreqaiExampleStrategy.py:263: error: Argument 2 to "reduce" has incompatible type "list[Series[builtins.bool]]"; expected "Iterable[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]]"  [arg-type]
- freqtrade/templates/FreqaiExampleStrategy.py:267: error: Unsupported left operand type for & ("tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]")  [operator]
- freqtrade/templates/FreqaiExampleStrategy.py:267: error: Argument 2 to "reduce" has incompatible type "list[Series[builtins.bool]]"; expected "Iterable[tuple[IndexOpsMixin[Any, Any] | Series[builtins.bool] | ndarray[tuple[Any, ...], dtype[numpy.bool[builtins.bool]]] | list[builtins.bool] | str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any] | Sequence[str | bytes | date | datetime | timedelta | <7 more items> | complex | integer[Any] | floating[Any] | complexfloating[Any, Any]] | slice[Any, Any, Any], ...]]"  [arg-type]

[ilevkivskyi]

Btw, I was thinking about batching SCCs in requests to workers, and I think there may be a simple conservative batching that (almost?) guarantees that we don't loose performance from less optimal bin packing on platforms with low socket overhead. I will play with this later, and if it will work, we can just do this on all platforms (as you know I am not a big fan of proliferation of platform-specific logic, even for optimizations).

@JukkaL just to clarify: I have no planned changes to this PR, this is ready for review (and I am still waiting for answers on questions I asked in the PR description).

[JukkaL]

Here's my review of the PR. I'll answer the questions in a separate comment.

[JukkaL]

Mismatch: 2/3

[JukkaL]

Typo: "bo error"

[JukkaL]

Did you check that overload bodies are only checked once? I think visit_overloaded_func_def in checker is now called twice, based on a quick experiment, when using parallel checking.

[ilevkivskyi]

Yes, the overload "header" is checked twice (and similar for decorators), while the actual body should be checked only once. I didn't fix this because: a) I am a bit worried fixing this may (subtly) break the daemon, b) performance impact from checking the header twice seems to be small.

However, thinking a bit more about this, we should at least try, either in this PR or in a follow-up.

[JukkaL]

It looks like write_cache() might return None? This seems to crash, and it's potentially related to this:

python -m mypy --cache-dir=/dev/null --num-workers=2 <some_file>.py

[ilevkivskyi]

Yeah, we should not allow disabling cache in parallel mode.

[JukkaL]

Narrowing of final global variables is not preserved in functions
anymore, see testNarrowingOfFinalPersistsInFunctions. Supporting
this will be tricky/expensive, it would require preserving binder
state at the point of each function definition, and restoring it
later. IMO this is a relatively niche edge case, and we can simply
"un-support" it (there is a simple workaround, add an assert in
function body). To be clear, there are no problems with a much
more common use of this feature: preserving narrowing in nested
functions/lambdas.

Agreed.

Support for --disallow-incomplete-defs in plugins doesn't work,
see testDisallowIncompleteDefsAttrsPartialAnnotations. I think
this should be not hard to fix (with some dedicated cleaner
support). I can do this in a follow-up PR soon.

Yeah, this can be fixed later.

Around a dozen incremental tests are skipped in parallel mode
because order of error messages is more unstable now (which is
expected). To be clear, we still group errors per module, but
order of modules is much less predictable now. If there are no
objections, I am going to ignore order of modules when comparing
errors in incremental tests in a follow-up PR.

Sounds good.

When inferred type variable variance is not ready, we fall back to
covariance, see
testPEP695InferVarianceNotReadyWhenNeeded. However, when
processing function/method bodies in a later phase, variance is
ready more often. Although this is an improvement, it creates an
inconsistency between parallel mode, and regular mode. I propose
to address this by making the two-phase logic default even without
parallel checking, see below.

Agreed, we don't want to have subtle apparently unrelated behavior changes
when enabling some flags, like we've had in daemon vs non-daemon runs.
Anyway, I'd like to make parallel checking enabled by default as soon as it's
stable and mature enough.

Finally, there are few edge cases with --local-partial-types when
behavior is different in parallel mode, see
e.g. testLocalPartialTypesWithGlobalInitializedToNone. Again the
new behavior is IMO clearly better. However, it again creates an
inconsistency with non-parallel mode. I propose to address this by
enabling two-phase (interface then implementation) checking
whenever --local-partial-types is enabled (globally, not
per-file), even without parallel checking. Since
--local-partial-types will be default behavior soon (and hopefully
the only behavior at some point), this will allow us to avoid
discrepancies between parallel and regular checking.

Yes, I like this idea.

[JukkaL]

Feel free to merge this when you feel it's ready. Some improvements can be done in follow-up PRs.