The source code and executable tool of PyRefcon is in
sub-directory pyrefcon.
Source code repository: https://github.com/Snape3058/PyRefcon
Build PyRefcon from source: Build LLVM with subproject
clang enabled by following the LLVM CMake build
instruction: https://releases.llvm.org/13.0.0/docs/CMake.html
tl;dr:
# Download source code:
$ git clone https://github.com/Snape3058/PyRefcon
# Build Clang
/path/to/PyRefcon$ mkdir build
/path/to/PyRefcon/build$ cmake ../llvm -DLLVM_ENABLE_PROJECTS=clang
/path/to/PyRefcon/build$ make clangThe detailed experimental results are provided in directory evaluation. We present statistics and
tables in CSV format, which can be clearly viewed with spreadsheet
editors like MS Excel, and LibreOffice Calc, or in Linux shell with
command column -t -s, input.csv | less -S.
evaluation/pyrefcon-reports.csv
shows the detailed information of each bug report generated by
PyRefcon, where the corresponding bug reports in HTML
format is presented in directory evaluation/pyrefcon-reports.
Besides, the additional false positives pruned by the model of monitors
are stored in directory evaluation/monitor-fp.
In the sheet, column project shows the name of the benchmark
instance. Column bug type presents the type of the reported
bug, where RL, UaR represent bug type Reference
Leak and Use-after-Release mentioned in Section 2.3. The
next four columns provide the location (file, function, and line number)
and path length of the report. The file name of the bug report is
presented in the next column, and the corresponding HTML bug report file
can be found from path
evaluation/pyrefcon-reports/<project>/report-xxxxxx.html.
Column result presents the result of manual bug report
revision, where TP and FP denote true-positive and
false-positive respectively. Column category provides the
category of the report that we assigned, where the type of true
positives are shown in Section 5.5, and reasons of false positives are
shown in Section 5.2. If the bug report is submitted to developers, the
link to the corresponding issue and its current status are presented in
column issue and submit. Otherwise, the reason why it
is not submitted is presented in column submit. In the last
column, the number of similar reports is presented. If the number is
greater than 1, it means there are redundant bug reports for it.
As mentioned in Section 5.1 and 5.3, we execute CpyChecker
on the same benchmark we used, where the information of the bug reports
generated by CpyChecker is presented in file evaluation/cpychecker-reports.csv.
In the sheet, all columns, except the last column, represent the same
meaning as the corresponding columns in file evaluation/pyrefcon-reports.csv.
The HTML bug report files presented in column report are stored
in path
evaluation/cpychecker-reports/<project>/xxxxxx.html.
And the last column shows whether the report is a unique report
that PyRefcon does not report, or it is a common
report that PyRefcon also reports. The common true positives
comparison of PyRefcon and CpyChecker is shown in
Figure 9a.
In Section 5.3, we also mentioned the literal comparison with
Pungi and RID through their comparisons with
CpyChecker. The detailed information of this comparison shown
in Figure 9b is presented in file evaluation/pungi-and-rid.html.
In section 5.4, we mentioned about average time and memory
consumption per kilo line of code. The average data is computed based on
the measurement on every translation unit, which is shown in file evaluation/time-and-memory-file.csv.
The first two columns in the sheet present the project name and the main file of the translation unit. The following ten columns denote the time (in seconds) and memory (in KiB) consumption of five executions. And the average of the five executions are shown in the last two columns for time and memory.
Besides, in Table 1 on page 7, we mentioned measured time consumption
and the estimated upper bound of memory consumption of each project
under a concurrency of 16 analyzer instances. The intermediate data of
execution time and measurement procedure of memory are illustrated in
file evaluation/time-and-memory-project.html
The bug reports generated in recent checks after revision of ASE 2023
on non-benchmark projects are archived in bug-reports.