This is awkward to track... Probably it would be best done by tracking
the ratio of unique IDs to lines as a floating point and plot them and
see what a best fit distribution curve looks like. But none of the
histogram printing or stats tracking stuff already in use here makes it
easy to do any of that...
So this does what I hope is a reasonable rough approximation by counting
the ceiling of unique identifiers per 10 lines of code, and plotting
that discreet histogram. Shape of the histogram is exactly what I would
expect: one centered distribution, vaguely normal looking. And the
center for a bunch of different codebases, including our toolchain, is
exactly at 5, which would mean 0.5 unique IDs per line. And the
distribution is pretty reliably bounded above by 10 or 1 unique ID per
line. Which almost seems to clean to be true? Slightly worried about
confirmation bias making me think this code is working because the
results look so pretty.
Here is the output for the toolchain:
```
## Unique IDs per 10 lines ## (median: 6)
2 ids [ 2] █▎
3 ids [19] ████████████▎
4 ids [32] ████████████████████▋
5 ids [55] ███████████████████████████████████▌
6 ids [62] ████████████████████████████████████████
7 ids [44] ████████████████████████████▍
8 ids [22] ██████████████▎
9 ids [11] ███████▏
10 ids [ 7] ████▌
11 ids [ 2] █▎
```
And here is the output for llvm-project/*/{lib,include} (to avoid
tests):
```
# Unique IDs per 10 lines ## (median: 5)
1 ids [ 29] ▍
2 ids [ 282] ███▊
3 ids [1492] ███████████████████▉
4 ids [2674] ███████████████████████████████████▌
5 ids [3011] ████████████████████████████████████████
6 ids [2267] ██████████████████████████████▏
7 ids [1549] ████████████████████▋
8 ids [ 817] ██████████▉
9 ids [ 301] ████
10 ids [ 98] █▎
11 ids [ 61] ▊
12 ids [ 50] ▋
13 ids [ 25] ▍
14 ids [ 33] ▌
15 ids [ 14] ▏
16 ids [ 15] ▎
17 ids [ 9] ▏
18 ids [ 8] ▏
19 ids [ 12] ▏
20 ids [ 15] ▎
21 ids [ 3]
22 ids [ 8] ▏
23 ids [ 3]
24 ids [ 3]
25 ids [ 6] ▏
26 ids [ 0]
27 ids [ 2]
28 ids [ 0]
29 ids [ 0]
30 ids [ 3]
31 ids [ 1]
32 ids [ 1]
```
Python [PEP-585](https://peps.python.org/pep-0585/) replaces a number of
`typing` module types with built-in equivalents and `collections.abc`
versions as of Python 3.9, with the aim of eventually removing the
`typing` module versions of these classes altogether. Since the minimum
required version of Python listed in the [Contribution Tools
document](https://github.com/carbon-language/carbon-lang/blob/trunk/docs/project/contribution_tools.md#main-tools)
is 3.9, the type hints in the various python files in the repo can be
updated to this style of type hint without a need for backwards
compatibility.
Feel free to close if this isn't a desired change at this time!
---------
Co-authored-by: Jon Ross-Perkins <jperkins@google.com>
This is pretty rough both in code-quality and what it tries to cover.
The coverage approximation is probably fine as it mostly just gives some
vague idea of the distributions we should aim at when evaluating
performance. We still want performance with unusual distributions to be
good.
Happy to improve the code quality if folks want, and especially with
suggestions on what would help. For this kind of quick scripting, it
seemed "fine" to me.
This is a rough script that uses regexes to do a simple scan of source
code and extract some basic source code statistics. Things like column
width, comment line density, identifier lengths and densities.
After scanning, it prints out both raw stats and in a few cases renders
a quick histogram to the terminal to help visualize a relevant
distribution.
I threw this together pretty quickly, and this is an area of Python I
have very limited familiarity with, so happy to have any suggestions for
how to better approach this.
Jon Ross-Perkins
Chandler Carruth
Jack McCluskeyandJon Ross-Perkins