[std-proposals] Modules and preambule, just why?

[m***@gmail.com]

Hi all,

I've got a question about modules since some times now.
The way modules have been done, they require the sources files to declare
which module they implements and what they exports into a preambule.
To be honest, I don't really get why this was necessary.
Moreover, i found some drawbacks with it:
- It makes sources files dependent on which modules they will be shipped in.
- Not only the sources does declare the module they belongs to, but they
also need to be put together in the build system (may it be Makefiles,
cmake, ...). This goes against the DRY concept since it cause duplication
of information.
- There is a concept of 'module interface unit' (and even a 'primary module
interface unit'!). As far as I understand it, it is almost the same that
old .h files, thus providing no real benefit over the latter when writing
new code.

So, my question is why was this preambule necessary? Can't we avoid it?
Also there seems to be a paper which indicate that preambules are
unnecessary
(http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1299r0.html).
What's the status about this?

Thanks,
Masse Nicolas.

[Klaim - Joël Lamotte]

The way modules have been done, they require the sources files to declare which module they implements and what they exports into a preambule.
To be honest, I don't really get why this was necessary.

My understanding is that the main reason is that it helps tools know
the dependencies without having to parse the rest of the C++ syntax.
Build systems will have to check every source file in a first pass to
quickly decide what have to be built, by making a dependency graph.
This pass is simple if the information is easily locatable and with
always a strict subset of possible C++ syntax.

(also it's similar to other module system from other languages, but
that's not a reason).

- It makes sources files dependent on which modules they will be shipped in.

I'm not sure I understand this point.
Isn't it the point of modules to associate a module name to sources?

- Not only the sources does declare the module they belongs to, but they also need to be put together in the build system (may it be Makefiles, cmake, ...). This goes against the DRY concept since it cause duplication of information.

I don't understand where would be a duplicate information. The module
names exist only in sources, not in build system's project description
files (at least in any decent build system).

- There is a concept of 'module interface unit' (and even a 'primary module interface unit'!). As far as I understand it, it is almost the same that old .h files, thus providing no real benefit over the latter when writing new code.

Of course it provides benefit! It's like saying that a header file and
a module is similar, but it's really not. For example, your header
will leak it's names into user's user's code. Modules will not by
default.
The point of these interface unit is to allow you to not change the
interface but change the implementation without the user aving to
recompile their code, even if all the module code is defined in one
file.

So, my question is why was this preambule necessary? Can't we avoid it?

I'm not an expert, but I believe it is necessary if we want something
that is easy to implement and works well with human readers.

Also there seems to be a paper which indicate that preambules are unnecessary (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1299r0.html). What's the status about this?

This paper will be published very soon, but you can ask the author for
a copy. Here is the revision 3's intro copy/pasted:

---
Module Preamble is Unnecessarily Fragile

Nathan Sidwell

The ATOM proposal introduced the module preamble. The merged document
modified that concept
for increased flexibility. The rules determining the extent of the
preamble have been modified, but it
remains fragile. We should consider making the preamble more robust.
TL;DR: This was presented to EWG at the San Diego’18 meeting. It was
accepted and is merged to
p1103.

----

As you can see, it's a fix of the preamble, not a removal.

A. Joël Lamotte

[m***@gmail.com]

Hi,

Post by m***@gmail.com
The way modules have been done, they require the sources files to

declare which module they implements and what they exports into a
preambule.

Post by m***@gmail.com
To be honest, I don't really get why this was necessary.

My understanding is that the main reason is that it helps tools know
the dependencies without having to parse the rest of the C++ syntax.
Build systems will have to check every source file in a first pass to
quickly decide what have to be built, by making a dependency graph.
This pass is simple if the information is easily locatable and with
always a strict subset of possible C++ syntax.

2 things here:
- The dependency graph could be quite hard to compute and will require to
parse every source files. While this is viable on small projects, this
could (and probably will) be a problem on large ones.
- So far, dependency have always been computed from the build system
itself, by specify which files are part of a library, and what other
libraries are needed to create each one of them.
Basically this is the opposite approach where this information is described
in the build system and is not know by the sources files. And so far, I do
believe this model works well and doesn't require to be changed.

Also, the way you phrase it seem to indicate we're modifying the language
to help other tools (not compilers) to deal with it. I hope it's not the
case.

(also it's similar to other module system from other languages, but

that's not a reason).

Agreed, it's not a reason.

Post by m***@gmail.com
- It makes sources files dependent on which modules they will be shipped

in.
I'm not sure I understand this point.
Isn't it the point of modules to associate a module name to sources?

I don't think so.
I think the point here is how to export the functions, global variables,
... in a more efficient way than using header files (eg: by avoiding to
parse and reparse the same code every times an header is included).
Note that after reading
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4047.pdf, it seems
they also speaks about "componentization", but i'm usure the current
proposal is the best way to achieve it (exporting or not a symbols can for
example already be specified using the visibility attribute when using
clang/gcc).

Post by m***@gmail.com
- Not only the sources does declare the module they belongs to, but they

also need to be put together in the build system (may it be Makefiles,
cmake, ...). This goes against the DRY concept since it cause duplication
of information.
I don't understand where would be a duplicate information. The module
names exist only in sources, not in build system's project description
files (at least in any decent build system).
As said above, this is basically the opposite approach. By the way why do

we need to put your files together in the source code inside a module while
we already put them inside the same library or executable at the build
system level?

- There is a concept of 'module interface unit' (and even a 'primary
module interface unit'!). As far as I understand it, it is almost the same
that old .h files, thus providing no real benefit over the latter when
writing new code.
Of course it provides benefit! It's like saying that a header file and
a module is similar, but it's really not. For example, your header
will leak it's names into user's user's code. Modules will not by
default.

The point of these interface unit is to allow you to not change the

interface but change the implementation without the user aving to
recompile their code, even if all the module code is defined in one
file.

The interface/implementation separation is already there when using header
files, so what you describe here seems to me already possible.
I'm not sure to get your point here.

Post by m***@gmail.com
So, my question is why was this preambule necessary? Can't we avoid it?

I'm not an expert, but I believe it is necessary if we want something
that is easy to implement and works well with human readers.

Post by m***@gmail.com
Also there seems to be a paper which indicate that preambules are

unnecessary (
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1299r0.html).
What's the status about this?
This paper will be published very soon, but you can ask the author for
---
Module Preamble is Unnecessarily Fragile
Nathan Sidwell
The ATOM proposal introduced the module preamble. The merged document
modified that concept
for increased flexibility. The rules determining the extent of the
preamble have been modified, but it
remains fragile. We should consider making the preamble more robust.
TL;DR: This was presented to EWG at the San Diego’18 meeting. It was
accepted and is merged to
p1103.
----
As you can see, it's a fix of the preamble, not a removal.

Yes, unfortunately.

A. Joël Lamotte

Thanks for your answer,
Masse Nicolas.

[Klaim - Joël Lamotte]

Hi,

Post by Klaim - Joël Lamotte

The way modules have been done, they require the sources files to declare which module they implements and what they exports into a preambule.
To be honest, I don't really get why this was necessary.

My understanding is that the main reason is that it helps tools know
the dependencies without having to parse the rest of the C++ syntax.
Build systems will have to check every source file in a first pass to
quickly decide what have to be built, by making a dependency graph.
This pass is simple if the information is easily locatable and with
always a strict subset of possible C++ syntax.

- The dependency graph could be quite hard to compute and will require to parse every source files. While this is viable on small projects, this could (and probably will) be a problem on large ones.

So far build2's main author reports no problem with this, as long as
the preamble is easilly parsable without having to have a complete C++
parser.
Also it seems to help making the rest of the compilation process faster.

I have some hello world using modules if you want to try them as a
starting point for converting a library and see if you see
improvements?

- So far, dependency have always been computed from the build system itself, by specify which files are part of a library, and what other libraries are needed to create each one of them.

That part do not change as module have nothing to do with the
dependency graph at the link level.

Basically this is the opposite approach where this information is described in the build system and is not know by the sources files. And so far, I do believe this model works well and doesn't require to be changed.

I think you're mixing the end result programs (exes, dlls/so, static
libraries) with modules. A C++ language module have nothing to do with
the actual generated binaries. It's code compartmentalization mainly.
You can have 10 modules in one so or in one exe, or in a
module-interface-only library (like a header-only library).

Also, the way you phrase it seem to indicate we're modifying the language to help other tools (not compilers) to deal with it. I hope it's not the case.

Modules add to the language, it doesn't "modify" the current one.
Tools are handicapped by the current include system.
Why would it be not the case? How do you want something like an IDE
extension trying to deduce where the words in the line you're pointing
to are from? They need to know the dependencies of the context.
Also the main tool that this helps is build systems, that can then
easily expose the dependency graph (of modules, not of binaries, which
they already do).

Post by Klaim - Joël Lamotte

- It makes sources files dependent on which modules they will be shipped in.

I'm not sure I understand this point.
Isn't it the point of modules to associate a module name to sources?

I don't think so.
I think the point here is how to export the functions, global variables, ... in a more efficient way than using header files (eg: by avoiding to parse and reparse the same code every times an header is included).

Here in the context of modules, "export" means "making these names
visible to the importer", it does not mean something like dllexport.
Just clarifying because the way you point this is weird to me, in this context.

Other than that I agree with you and it seems to do exactly that:
modules are compiled once. The preamble pre-reading to build the
modules dependency graph is trivial compared to what is needed to
parse the language for compilation.

Note that after reading http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n4047.pdf, it seems they also speaks about "componentization", but i'm usure the current proposal is the best way to achieve it (exporting or not a symbols can for example already be specified using the visibility attribute when using clang/gcc).

Modules do nothing to symbols. They componentize names.
If you want to export symbols, as in making a function in a .so/,dll
usable by another program, you stil have to add some dllexport/import
macros as it have nothing to do with modules.
I have an example here where I use some import/export macro
(symexport) in addition to a module export:
https://github.com/Klaim/build2-libhelloworld-exe-modularized/blob/master/libhelloworld-modules/helloworld-modules.mxx
(The syntax is the one implemented in VS 15.9 so it's Modules TS
syntax, not more recent Module proposal version).

Note that there is another proposal for a library API for helping with
plugins. But again this have nothing to do with modules.

Post by Klaim - Joël Lamotte

- Not only the sources does declare the module they belongs to, but they also need to be put together in the build system (may it be Makefiles, cmake, ...). This goes against the DRY concept since it cause duplication of information.

I don't understand where would be a duplicate information. The module
names exist only in sources, not in build system's project description
files (at least in any decent build system).

As said above, this is basically the opposite approach.

Again I think you're mixing binaries dependencies and modules (as in
code-only) dependencies.

By the way why do we need to put your files together in the source code inside a module while we already put them inside the same library or executable at the build system level?

The basic idea is that it depends on how you like to organize your
code. Modules are designed to not get in your way with this.
So if you want to have all the code in one module file (which then is
the module interface, with a public eported part, and maybe another
part not exported), nothing prevent you to do so.
If you want to splitt 1 Module into say 10 source files, you have
different ways to do so. The main thing is that at the end there is
only one file that defines what the module exports.
If you want to splitt your module in several modules then make one of
the module export the others, you can also to that.
If you want to have only the module interface containing exported
names, and all the other source module files having the definitions,
you can.
Etc. Organize as you want, the only constraint is that for one module,
there needs to be one file that will export the names for importers.
The details of how to do these is longer to explain but I think
several talks and papers can help with that.

Again, this have nothing to do with the binary linking part of the
process of building programs, so you can have 10 modules in a .so or
in a exe if you want.

Post by Klaim - Joël Lamotte

- There is a concept of 'module interface unit' (and even a 'primary module interface unit'!). As far as I understand it, it is almost the same that old .h files, thus providing no real benefit over the latter when writing new code.

Of course it provides benefit! It's like saying that a header file and
a module is similar, but it's really not. For example, your header
will leak it's names into user's user's code. Modules will not by
default.
The point of these interface unit is to allow you to not change the
interface but change the implementation without the user aving to
recompile their code, even if all the module code is defined in one
file.

The interface/implementation separation is already there when using header files, so what you describe here seems to me already possible.
I'm not sure to get your point here.

Ok, to clarify: it's kind of the same idea but done in a different way
which removes the issues with headers.
Among numerous issues, headers (assuming with only interfaces exposed
in them), cannot guarantee independence of what they expose.

#include "aaa.hpp" // void foo(Bar b);
#include "bbb.hpp" // #define Bar Kikoo

Can give a different code than

#include "bbb.hpp" // #define Bar Kikoo
#include "aaa.hpp" // void foo(Bar b);

Here both headers interract because in the end their content only
exist in the cpp including, so they impact each other.
You can try to make sure that your headers will always produce
equivalent code whatever the the other includes around (we call these
"well behaved" and there is a proposed mechanism
to help use them as if they were modules), but you are still dependent
on them and external ones because just one header could redefine any
word used in the following header included (mainly through macros).

With modules:

import aaa;
import bbb;

and

import bbb;
import aaa;

Are stictly equivalent, by design, whatever is define in these
modules. Importing one will not change the other as macros are simply
not imported, and only names/signatures can be exported.
Because this is only importing names from these modules in the set of
possible names that could be found when you try to use a name.

But you are right that in essence the module interface solve the same
problem (better) than having headers with only interfaces
declarations/definitions.

Post by Klaim - Joël Lamotte
As you can see, it's a fix of the preamble, not a removal.

Yes, unfortunately.

I'm not sure what is the problem with the preamble exactly? Did you
want to import from another place?

Thanks for your answer,

No problem. :)

Joël

[m***@gmail.com]

After reading your answer and thinking about it in the last days, I thing I
understand things a bit better now.
First I understand that module are far less related to the linking problem
than I first tough.
Also, I think I better understand how they will actually work in real world
application.

Still, one question remains for me:
If for example you wanna ship a library.
As of today, you generally with the library the headers so that people
using the library will be able to compile their code and make use of the
library you made.
But how does it work if you wanna use modules? do you ship a module file
together with the lib? And if so, how do you intend to generate this module
file?

Also note that even if I better understand the proposal now, my doubts
about it and the need of declaring the module explicitly remains.
In fact, I would have prefer an approach where the exported symbols are
deduced from the code instead of having to declare it explicitly.

Masse Nicolas.

[Ville Voutilainen]

After reading your answer and thinking about it in the last days, I thing I understand things a bit better now.
First I understand that module are far less related to the linking problem than I first tough.
Also, I think I better understand how they will actually work in real world application.
If for example you wanna ship a library.
As of today, you generally with the library the headers so that people using the library will be able to compile their code and make use of the library you made.
But how does it work if you wanna use modules? do you ship a module file together with the lib? And if so, how do you intend to generate this module file?

Yes, you ship the module interface with your lib. You compile the
module for the target implementation you want to support.

Also note that even if I better understand the proposal now, my doubts about it and the need of declaring the module explicitly remains.
In fact, I would have prefer an approach where the exported symbols are deduced from the code instead of having to declare it explicitly.

Deduced how? The problem we have today with headers is that everything
in a header needs to be parsed, whether part of the actual
interface of the library or not. Modules give us control over what
things are part of the interface and what things are not.

[Tom Honermann]

Post by Ville Voutilainen

After reading your answer and thinking about it in the last days, I thing I understand things a bit better now.
First I understand that module are far less related to the linking problem than I first tough.
Also, I think I better understand how they will actually work in real world application.
If for example you wanna ship a library.
As of today, you generally with the library the headers so that people using the library will be able to compile their code and make use of the library you made.
But how does it work if you wanna use modules? do you ship a module file together with the lib? And if so, how do you intend to generate this module file?

Yes, you ship the module interface with your lib. You compile the
module for the target implementation you want to support.

To be clear, I believe Ville means that you ship the module interface
unit source code (not a binary module artifact as they are not portable)
and you link any object files produced by compiling the module interface
unit in your compiled library for your supported platforms.

Tom.

Post by Ville Voutilainen

Also note that even if I better understand the proposal now, my doubts about it and the need of declaring the module explicitly remains.
In fact, I would have prefer an approach where the exported symbols are deduced from the code instead of having to declare it explicitly.

Deduced how? The problem we have today with headers is that everything
in a header needs to be parsed, whether part of the actual
interface of the library or not. Modules give us control over what
things are part of the interface and what things are not.

[Ville Voutilainen]

Post by Tom Honermann

Post by Ville Voutilainen

After reading your answer and thinking about it in the last days, I thing I understand things a bit better now.
First I understand that module are far less related to the linking problem than I first tough.
Also, I think I better understand how they will actually work in real world application.
If for example you wanna ship a library.
As of today, you generally with the library the headers so that people using the library will be able to compile their code and make use of the library you made.
But how does it work if you wanna use modules? do you ship a module file together with the lib? And if so, how do you intend to generate this module file?

Yes, you ship the module interface with your lib. You compile the
module for the target implementation you want to support.

To be clear, I believe Ville means that you ship the module interface
unit source code (not a binary module artifact as they are not portable)
and you link any object files produced by compiling the module interface
unit in your compiled library for your supported platforms.

I think I mean that you have the choice of shipping 0-N module
interface artifacts and/or the source code.
Good catch, though, Tom. However, I think the "you compile the module
for the target implementation you
want to support" is 60% correct. (Yeah, I pulled that 60% number out
of my sleeve, it doesn't really mean anything).
For the remaining 40% (the number is, again, completely artificial),
you might want to ship something else, for example the source code.

[Tom Honermann]

Post by Ville Voutilainen

Post by Tom Honermann

Post by Ville Voutilainen

After reading your answer and thinking about it in the last days, I thing I understand things a bit better now.
First I understand that module are far less related to the linking problem than I first tough.
Also, I think I better understand how they will actually work in real world application.
If for example you wanna ship a library.
As of today, you generally with the library the headers so that people using the library will be able to compile their code and make use of the library you made.
But how does it work if you wanna use modules? do you ship a module file together with the lib? And if so, how do you intend to generate this module file?

Yes, you ship the module interface with your lib. You compile the
module for the target implementation you want to support.

To be clear, I believe Ville means that you ship the module interface
unit source code (not a binary module artifact as they are not portable)
and you link any object files produced by compiling the module interface
unit in your compiled library for your supported platforms.

I think I mean that you have the choice of shipping 0-N module
interface artifacts and/or the source code.

Hmm, Gaby has stated many times that .ifc files are not intended for
distribution.  Clang, from what I understand, is not promising
compatibility for module artifacts across major releases (which makes
compatibility across Xcode and LLVM Clang questionable). Gcc currently
ties the module artifact to the svn revision.  I am very much under the
impression that shipping module artifacts (without the corresponding
module interface unit source) will not be a viable choice in practice
(at least, not for anyone other than compiler implementors).

Post by Ville Voutilainen
Good catch, though, Tom. However, I think the "you compile the module
for the target implementation you
want to support" is 60% correct. (Yeah, I pulled that 60% number out
of my sleeve, it doesn't really mean anything).
For the remaining 40% (the number is, again, completely artificial),
you might want to ship something else, for example the source code.

I meant that, if you are shipping a pre-built library, then the portions
of the module interface unit that are not "interface" (e.g.,
non-template function definitions) probably need to be pre-compiled and
linked into the distributed library (which may have its own dependencies
on them) unless the expectation is that the recipient link objects
produced by compilation of module interface units into their respective
binaries (which seems to me a good recipe for ending up with duplicate
definition errors at link time).  I'm very curious to see how build
systems are going to end up managing this.

Tom.

[Klaim - Joël Lamotte]

Post by m***@gmail.com
If for example you wanna ship a library.
As of today, you generally with the library the headers so that people using the library will be able to compile their code and make use of the library you made.

You are focusing specifically on the shared-library case, ok. Just
note that it's not always the best solution for all libraries or
users.

Post by m***@gmail.com
But how does it work if you wanna use modules? do you ship a module file together with the lib? And if so, how do you intend to generate this module file?

Just ship a module interface unit that only contain the exported names
marked, if implemented separately, as exported/imported symbols too.
That's where it's interesting to have a module interface unit only
contain the exported names+symbols.

////////////////////////////////////////////////////////////////////////////
//// Module interface: mymodule.mxx

export module mymodule;

// MYMODULE_API does the export/import dance, or look at build2's
__symexport as another way to do this.

export namespace mymodule { // export all the names in this namespace scope

class Kikoo; // Name exported but not symbol import/export: we
don't want the user to use it's interface, just see it's name because
it's used in other interfaces

MYMODULE_API Kikoo& foo(); // name exported but also symbol import/export
MYMODULE_API void bar(Kikoo&); // name exported but also symbol
import/export


template<class T>
struct Lol { ... }; // Template, so no need to export symbols, it
might benefit from less compile time for the user though.

}

//////////////////////////////////////////////////
////// Module implementation : mymodule.cpp
module mymodule:impl; // If I remember correctly, that's a module
partition, not sure about the syntax

// Just implement everything here.
namespace mymodule{
class Kikoo { ... };

Kikoo& foo() { ... }
void bar(Kikoo&) { ... }

}

//////////////////////////////////////////////////////

I'm not sure of the exact syntax so bear with me, but basically once
you build that in a shared library, you just have to provide
mymodule.mxx to the user, with the shared library binary (and maybe a
.lib on Windows).
You can try that in the library example using build2+msvc I linked to,
just move the implementation in a separate module file with the same
module name but not exporting.
The test is setup to link with the library, shared mode by default, so
if the test work, it correctly only used the module interface and
linked to it.

Post by m***@gmail.com
Also note that even if I better understand the proposal now, my doubts about it and the need of declaring the module explicitly remains.

No problem with that, we can probably help clarifying or maybe you can
find a pain point before it gets standardized.

Post by m***@gmail.com
In fact, I would have prefer an approach where the exported symbols are deduced from the code instead of having to declare it explicitly.

As pointed before, modules aren't related with linking. Though once
they are available, we will finally have a clearer way to specify in
C++ the boundaries of a library.
With this, it might be possible later to help with the symbol
export/import, but that seems complicated as it imply specifying some
stuffs that are currently unspecified.

A. Joël Lamotte