The cause of the internal error was the following:
We have a generic in an unit ("A") which uses another unit ("B") in the implementation section and this other unit uses unit A in the interface section. Now the generic is specialized in the interface section of B. This leads to the problem that in unit A when it tries to load the globalsymtable of unit B that globalsymtable will be Nil, because parsing of the interface section is not yet finished. Thus the change in pgenutil.pas, specialization_init: if the unit is still "in_interface" the localsymtable needs to be used instead of the globalsymtable.
This doesn't necessarily lead to a compiling test though, as there is the following possibility:
Unit A contains a generic class/record (with methods) and uses unit B in the implementation section. This unit B also contains a generic class/record (with methods) and uses unit A in the implementation section. Both units contain a specialization of the other unit's generic outside of it's own generics (such that generate_specialization is fully triggered). Let's assume compilation starts with unit A and we reach the uses of unit B. Now compilation switches to unit B and completes as unit A is already registered and in compilation. The problem now is that the generic in unit A still contains unresolved forward declarations as the implementation section of A was not yet parsed which will lead to "forward declaration not solved" errors (Note: Delphi compiles this).
The solution to this is the following: if a generic is specialized from another unit which is not in state ms_compiled then the unit of the specialization needs to wait for the unit of the generic. So the specialization's unit adds itself into a list of waiting units of the generic's unit. Now inside "proc_unit" we need to check whether this module is waiting for other modules and if so avoid "finishing" the unit (which means generating the methods of the specialization, generating assembler code and ultimately freeing the scanner and PPU). Now when the generic's unit finishes we need to check whether other modules are waiting for it and finish them (of course it's a bit more complicated in reality, but that pretty much sums it up).
+ globstat.pas: Added an unit which handles the saving and restoring of the global state which was originally inside "parser.pas, compile" so that Don't Repeat Yourself (DRY) is respected.
* fmodule.pas, tmodule:
+ add fields to keep track of the units the module is waiting for and which modules are waiting for the module
+ add field for the saved global state (raw pointer to avoid circles)
+ add field for the state which is needed to finish the unit (raw pointer to avoid circles)
+ move the code which was used in "parser.pas, compile" after a module was successfully compiled to the new virtual method "end_of_parsing"
+ fppu.pas, tppumodule.end_of_parsing:
free the ppufile here
* pgenutil.pas:
+ add new procedure "maybe_add_waiting_unit" which adds the specialization's unit to the waiting list of the generic if that unit is not yet compiled
* generate_specialization: call the new function when we add a new (true) specialization
* specialization_init: instead of not adding implementation units at all check whether the unit is still parsing the interface section and add the localsymtable in that case
* pmodules.pas:
* change "proc_unit" to a function which returns "true" if the unit was already finished (no need to wait for other units)
+ move the code from "proc_unit" from "generate_specialization_procs" on to a new procedure "finish_unit" which
* this procedure is either called immediately in "proc_unit" if the unit does not need to wait for other units or from "finish_unit" itself if a unit that is waiting for the given unit does no longer wait for another module (special care is taken in proc_unit to avoid circles)
* parser.pas, compile:
* correctly handle the case if an unit is not finished
* use the new global state functionality from globstat.pas
* pay special attention when calling "set_current_module" (see comment at that call)
+ add tests from 22160
+ add test for above mentioned "diamond" case
git-svn-id: trunk@22452 -
* pscanner.pas, tscannerfile.replaytoken: remove the checks for "sources_avail" and always update the read file/line information
* fppu.pas, tppumodule.readsourcefiles: always load at least the filenames which were used when compiling the unit from the ppu file, so that file/line information will be correctly shown, but don't touch the "sources_avail" flag of the tppumodule
git-svn-id: trunk@21657 -
that deal with paths/filenames with TPathStr (= ansistring) to prevent
cutting off long paths (no change in speed when compiling the compiler,
1% extra memory usage)
git-svn-id: trunk@21120 -
o support for the new codepage-aware ansistrings in the jvm branch
o empty ansistrings are now always represented by a nil pointer rather than
by an empty string, because an empty string also has a code page which
can confuse code (although this will make ansistrings harder to use
in Java code)
o more string helpers code shared between the general and jvm rtl
o support for indexbyte/word in the jvm rtl (warning: first parameter
is an open array rather than an untyped parameter there, so
indexchar(pcharvar^,10,0) will be equivalent to
indexchar[pcharvar^],10,0) there, which is different from what is
intended; changing it to an untyped parameter wouldn't help though)
o default() support is not yet complete
o calling fpcres is currently broken due to limitations in
sysutils.executeprocess() regarding handling unix quoting and
the compiler using the same command lines for scripts and directly
calling external programs
o compiling the Java compiler currently requires adding ALLOW_WARNINGS=1
to the make command line
git-svn-id: branches/jvmbackend@20887 -
for the current unit and all types/routines declared in it. The
unit itself becomes a member of this namespace as well, so in
case it's called unit1, it will be x.y.z.unit1, and type tclass
declared in it will be x.y.z.tclass. Only used for the JVM
target currently
git-svn-id: branches/jvmbackend@18436 -
in Windows, Emx and OS2 targets.
This fixes test failures: test/library/tlib1b.pp and lib2b.pp
link unit TLinker.AddImportSymbol: Added symmangledname parameter.
ogbase unit TImportSymbol.Create: Add AMangledName parameter.
fmodule unit TModule.AddExternalImport: Add symmangledname parameter.
fppu unit: Also put mangled name string for imported symbol.
ppu unit: Increase PPUVersion
utils/ppudump.pp: Adapt to PPU change above.
systems/T_OS units: Use ImportSymbol.MangledName property to create
import libraries or .idata sections.
git-svn-id: trunk@17804 -
* TCmdStr changed into an ansistring
* command line parameter handling uses now ansistrings
* message handling parameters are now ansistrings
git-svn-id: trunk@17469 -
of a pre-compiled unit are available on tf_p_ext_support systems
(patch by Aleksa Todorovic, mantis #17907)
* fixed indentation/flow of this and a similar check elsewhere
git-svn-id: trunk@16328 -
dependent units, but the ppu that caused the current unit to be loaded
-> last comment of r14505 was wrong, and that change has been reverted
git-svn-id: trunk@14506 -
o keep track of a global defsgeneration counter that's increased every time
a unit has finished compiling its interface or implementation
o give each ppumodule the current defsgeneration number when its compiled
o only re-resolve a unit if its defsgeneration number is smaller than that
of the unit that wants it to be re-resolved
* fixed an error in the re-resolving logic added last time: re-resolve
units that depend on the current unit rather than those that also depend
on the parent unit
git-svn-id: trunk@14505 -
units have changed, because these can influence the code of the
current unit in case method signatures changed (mantis #13840)
+ manual test
* fixed some recompilation crashes related to WPO info
* clarified the unit_u_add_depend_to message
git-svn-id: trunk@14503 -
svn+ssh://jonas@svn.freepascal.org/FPC/svn/fpc/branches/wpo
........
r11878 | jonas | 2008-10-11 02:25:18 +0200 (Sat, 11 Oct 2008) | 19 lines
+ initial implementation of whole-program optimisation framework
+ implementation of whole-program devirtualisation
o use:
a) generate whole-program optimisation information (no need
to completely compile the program and all of its units
with -OW/-FW, only the main program is sufficient)
fpc -OWdevirtcalls -FWmyprog.wpo myprog
b) use it to optimise the program
fpc -B -Owdevirtcalls -Fwmyprog.wpo myprog
(the -B is not required, but only sources recompiled during
the second pass will actually be optimised -- if you want,
you can even rebuild the rtl devirtualised for a particular
program; and these options can obviously also be used
together with regular optimisation switches)
o warning:
- there are no checks yet to ensure that you do not use
units optimised for a particular program with another
program (or with a changed version of the same program)
........
r11881 | jonas | 2008-10-11 19:35:52 +0200 (Sat, 11 Oct 2008) | 13 lines
* extracted code to detect constructed class/object types from
tcallnode.gen_vmt_tree into its own method to avoid clutter
* detect x.classtype.create constructs (with classtype = the
system.tobject.classtype method), and treat them as if a
"class of x" has been instantiated rather than a
"class of tobject". this required storing the instantiated
classrefs in their own array though, because at such a
point we don't have a "class of x" tdef available (so
now "x", and all other defs instantiated via a classref,
are now stored as tobjectdefs in a separate array)
+ support for devirtualising class methods (including
constructors)
........
r11882 | jonas | 2008-10-11 20:44:02 +0200 (Sat, 11 Oct 2008) | 7 lines
+ -Owoptvmts whole program optimisation which replaces vmt entries
with method names of child classes in case the current class'
method can never be called (e.g., because this class is never
instantiated). As a result, such methods can then be removed
by dead code removal/smart linking (not much effect for either
the compiler, lazarus or a trivial lazarus app though).
........
r11889 | jonas | 2008-10-12 14:29:54 +0200 (Sun, 12 Oct 2008) | 2 lines
* some comment fixes
........
r11891 | jonas | 2008-10-12 18:49:13 +0200 (Sun, 12 Oct 2008) | 4 lines
* fixed twpofilereader.getnextnoncommentline() when reusing a previously
read line
* fixed skipping of unnecessary wpo feedback file sections
........
r11892 | jonas | 2008-10-12 23:42:43 +0200 (Sun, 12 Oct 2008) | 31 lines
+ symbol liveness wpo information extracted from smartlinked programs
(-OW/-Owsymbolliveness)
+ use symbol liveness information to improve devirtualisation (don't
consider classes created in code that has been dead code stripped).
This requires at least two passes of using wpo (first uses dead code
info to locate classes that are constructed only in dead code,
second pass uses this info to potentially further devirtualise).
I.e.:
1) generate initial liveness and devirtualisation feedback
fpc -FWtt.wpo -OWall tt.pp -Xs- -CX -XX
2) use previously generated feedback, and regenerate new feedback
based on this (i.e., disregard classes created in dead code)
fpc -FWtt-1.wpo -OWall -Fwtt.wo -Owall tt.pp -Xs- -CX -XX
3) use the newly generated feedback (in theory, it is possible
that even more opportunities pop up afterwards; you can
continue until the program does not get smaller anymore)
fpc -Fwtt-1.wpo -Owall tt.pp -CX -XX
* changed all message() to cgmessage() calls so the set codegenerror
* changed static fsectionhandlers field to a regular field called
fwpocomponents
* changed registration of wpocomponents: no longer happens in the
initialization section of their unit, but in the InitWpo routine
(which has been moved from the woinfo to the wpo unit). This way
you can register different classes based on the target/parameters.
+ added static method to twpocomponentbase for checking whether
the command line parameters don't conflict with the requested
optimisations (e.g. generating liveness info requires that
smartlinking is turned on)
+ added static method to twpocomponentbase to request the
section name
........
r11893 | jonas | 2008-10-12 23:53:57 +0200 (Sun, 12 Oct 2008) | 3 lines
* fixed comment error (twpodeadcodeinfo keeps a list of live,
not dead symbols)
........
r11895 | jonas | 2008-10-13 00:13:59 +0200 (Mon, 13 Oct 2008) | 2 lines
+ documented -OW<x>, -Ow<x>, -FW<x> and -Fw<x> wpo parameters
........
r11899 | jonas | 2008-10-14 22:14:56 +0200 (Tue, 14 Oct 2008) | 2 lines
* replaced hardcoded string with objdumpsearchstr constant
........
r11900 | jonas | 2008-10-14 22:15:25 +0200 (Tue, 14 Oct 2008) | 2 lines
* reset wpofeedbackinput and wpofeedbackoutput in wpodone
........
r11901 | jonas | 2008-10-14 22:16:07 +0200 (Tue, 14 Oct 2008) | 2 lines
* various additional comments and comment fixes
........
r11902 | jonas | 2008-10-15 18:09:42 +0200 (Wed, 15 Oct 2008) | 5 lines
* store vmt procdefs in the ppu files so we don't have to use a hack to
regenerate them for whole-program optimisation
* fixed crash when performing devirtualisation optimisation on programs
that do not construct any classes/objects with optimisable vmts
........
r11935 | jonas | 2008-10-19 12:24:26 +0200 (Sun, 19 Oct 2008) | 4 lines
* set the vmt entries of non-class virtual methods of not instantiated
objects/classes to FPC_ABSTRACTERROR so the code they refer to can
be thrown away if it is not referred to in any other way either
........
r11938 | jonas | 2008-10-19 20:55:02 +0200 (Sun, 19 Oct 2008) | 7 lines
* record all classrefdefs/objdefs for which a loadvmtaddrnode is generated,
and instead of marking all classes that derive from instantiated
classrefdefs as instantiated, only mark those classes from the above
collection that derive from instantiated classrefdefs as
instantiated (since to instantiate a class, you have to load its vmt
somehow -- this may be broken by using assembler code though)
........
r12212 | jonas | 2008-11-23 12:26:34 +0100 (Sun, 23 Nov 2008) | 3 lines
* fixed to work with the new vmtentries that are always available and
removed previously added code to save/load vmtentries to ppu files
........
r12304 | jonas | 2008-12-05 22:23:30 +0100 (Fri, 05 Dec 2008) | 4 lines
* check whether the correct wpo feedback file is used in the current
compilation when using units that were compiled using wpo information
during a previous compilation run
........
r12308 | jonas | 2008-12-06 18:03:39 +0100 (Sat, 06 Dec 2008) | 2 lines
* abort compilation if an error occurred during wpo initialisation
........
r12309 | jonas | 2008-12-06 18:04:28 +0100 (Sat, 06 Dec 2008) | 3 lines
* give an error message instead of crashing with an io exception if the
compiler is unable to create the wpo feedback file specified using -FW
........
r12310 | jonas | 2008-12-06 18:12:43 +0100 (Sat, 06 Dec 2008) | 3 lines
* don't let the used wpo feedback file influence the interface crc (there's
a separate check for such changes)
........
r12316 | jonas | 2008-12-08 19:08:25 +0100 (Mon, 08 Dec 2008) | 3 lines
* document the format of the sections of the wpo feedback file inside the
feedback file itself
........
r12330 | jonas | 2008-12-10 22:26:47 +0100 (Wed, 10 Dec 2008) | 2 lines
* use sysutils instead of dos to avoid command line length limits
........
r12331 | jonas | 2008-12-10 22:31:11 +0100 (Wed, 10 Dec 2008) | 3 lines
+ support for testing whole program optimisation tests (multiple
compilations using successively generated feedback files)
........
r12332 | jonas | 2008-12-10 22:31:40 +0100 (Wed, 10 Dec 2008) | 2 lines
+ whole program optimisation tests
........
r12334 | jonas | 2008-12-10 22:38:07 +0100 (Wed, 10 Dec 2008) | 2 lines
- removed unused local variable
........
r12339 | jonas | 2008-12-11 18:06:36 +0100 (Thu, 11 Dec 2008) | 2 lines
+ comments for newly added fields to tobjectdef for devirtualisation
........
r12340 | jonas | 2008-12-11 18:10:01 +0100 (Thu, 11 Dec 2008) | 2 lines
* increase ppu version (was no longer different from trunk due to merging)
........
git-svn-id: trunk@12341 -
the code using {$pascalmainname x} + storing it in the ppu file
(and give a warning if it's overridden multiple times + test)
git-svn-id: trunk@10406 -
* fixed some (harmless) ptrint warnings
- removed some commented code in agppcmpw
* added one missing field in a typed constant in cp1251
git-svn-id: trunk@8081 -
It is done in the following way:
When unit is compiled, .rc file are compiled to .res and list of unit's resource files is stored in .ppu
Before final linking all program's .res files are collected into global .res file (.res files are easily concatenated).
Then this global .res files is compiled to single .or file, which is linked to executable.
As a result global resource index is created and the problem is fixed.
Old resource processing behavior still supported when tresinfo.rcbin is not set for target.
New resource processing is activated for windows and linux. Cross compiled windres can be used to compile .rc files on linux.
git-svn-id: trunk@7515 -
* new set_current_module function that sets the current_module and
all related variables. Also closes scanner files if required, but
that might still need some optimization to prevent closing/opening
files too often
git-svn-id: trunk@7428 -
* symtables based on TFPHashObjectList and TFPObjectList
* rename torddef.typ to torddef.ordtype
* rename tfloatdef.typ to tfloatdef.floattype
* rename tdef.deftype to tdef.typ
* remove obsolete browser code, browcol is kept so the ide
can still be compiled
git-svn-id: trunk@5192 -
