Jeremy Pack's Coding Blog

Questions

2010-04-15T16:44:00.000-07:00

If you have questions on Boost.Extension, feel free to contact me:

Contact Jeremy Pack

Hanging Chain with Varying Weights

2010-03-26T16:30:00.000-07:00

Here's another hanging chain scenario. The red links are a little longer and much heavier.

The Hanging Chain Problem - or Catenary

2010-03-25T16:50:00.000-07:00

Here's a video from some software I just wrote for the class I've been taking part time (you should never stop learning!)

Here's my description of the problem from my report:

Consider a chain hung from the ceiling. How can we predict its
shape as it hangs? Each link in the chain will naturally fall until it
is pulled back by neighboring links. As the links fall, the potential
energy of the chain decreases until the chain has reached it’s ﬁnal
shape.
If we minimize the potential energy of the whole chain, while keep-
ing the links connected, we can ﬁnd this ﬁnal shape. The most obvious
method to solve this problem would be to slowly attempt to move the
links of the chain downward without unlinking them. Each link would
move a little bit at a time, eventually reaching its ﬁnal position.
However, such a method could be very slow - especially for large
chains. First of all, the actual movement of each link is restricted
by the neighboring links, meaning that only inﬁnitesimal movements
could be made at a time. In addition, moving any link will aﬀect every
other link in the chain.

I'll add a few details of my solution once everyone else has turned their projects in.

Run-time Compilation - Performance

2010-01-29T09:06:00.000-08:00

I put together an example of compiling code generated at run-time, and then loading it using Extension:

https://svn.boost.org/trac/boost/browser/sandbox/libs/extension/examples/runtime_compilation

I'm preparing some experiments to compare the performance of numerical code compiled in advance to that compiled at run-time. There will be three test cases involving complex matrix operations:

The matrix sizes is known at compile-time, and can be used by the compiler for optimization. The matrix computations are done in the main binary.
The matrix sizes are not known at compile-time. The matrix computations are done in the main binary using algorithms that work for arbitrarily-sized matrices.
The matrix sizes are not known when the binary is compiled. When the binary loads a matrix, it will generate code on the fly to process the matrix. It will then compile the code into a shared library, load the shared library, and run the algorithm.

My hypothesis is that it will be possible to make the runtime-generated code faster than the arbitrarily-sized matrix code, for the following reasons:

The compiler will be able to make certain optimizations that it can only do when it knows the size of the arrays being looped over.
I'll be able to use fewer variables and more constants in the compiled code.

I expect it to be slower than the specialized pre-compiled code for two reasons though:

Code in a shared library usually runs more slowly than code in the binary itself. One reason for this is that shared libraries do not know which address space they will be running at during run-time, and can't hardcode as many pointers and such as an executable. (Note that operating system libraries often have an optimization to avoid this problem, by using reserved address space)
Even if it did run as fast, it has to be compiled and loaded first.

Any opinions? I hope to post preliminary results in the next couple weeks or so.

Explanation

2010-01-29T08:52:00.000-08:00

I consider Extension basically complete - quite a number of people are using it in projects. The main issue holding me back from submitting it for review by the Boost community is that I wanted to make the Reflection library part of Extension in a natural way, and I've been struggling at finding a balance between flexibility, performance and readability for the Reflection API. Also, of course, the Boost library review process is a bit painful - I'd need to block out at least 100 hours of my time to make sure the documentation, API, code, tests etc. were correct and easy to follow.

My previous post shouldn't be considered as much an obituary as an explanation of the slow progress lately. As I mentioned, I haven't been able to make much progress on the Reflection/Extension integration and new Extension features partially because it's difficult to design an API without having code that requires it.

If you find any features lacking in Extension, or aren't sure how to get the functionality you need, feel free to describe your use case to me - I can add the relevant functionality or documentation.

Also, always feel free to send patches. Extension is in the Boost sandbox:

https://svn.boost.org/trac/boost/browser/sandbox/libs/extension
https://svn.boost.org/trac/boost/browser/sandbox/boost/extension

https://svn.boost.org/trac/boost/browser/sandbox/libs/reflection

https://svn.boost.org/trac/boost/browser/sandbox/boost/reflection

Alternatives to Plugins

2010-01-28T20:17:00.000-08:00

I honestly haven't put much time into Boost.Extension and Boost.Reflection lately. My current work has no need for plugins of that type - and it's difficult to continue designing something without good current use cases to consider.

I've actually been surprised by the use cases many people propose to me in e-mail. Some people are looking for a full reimplementation of COM, and want plugins that can be compiled with a very different set of compiler options than the binary loading the plugin. In most cases, I felt like their use case did not justify such a complex solution.

So, do you really need C++ plugins? Consider the following:

Do you need C++ performance? Or would a Boost.Python solution of integrating plugins be more effective? Python is certainly easier to deploy.
Do you really need plugins? Or do you just need to compile your binary with different modules for different users?
Are you shipping your source code to users to compile? If so, they can just compile in the plugins instead of putting them in a shared library.

If C++ plugins are still desired, here are my recommendations:

Always compile the binary and plugins with the same compiler options.
If you really need different compiler options:

Consider building a distributed system, with the binary and the plugin communicating over network sockets or IPC.
Consider using a plain C API, with very simple types being passed between the binary and the plugin (even structs can be a problem, since they may have different sizes depending on compiler options).

I plan to start posting here about some of my current work on the performance of optimization algorithms. I'm building a framework to automatically instrument and test different linear and nonlinear algorithms. I also hope to get back to dabbling in OpenGL...

Resource Management Across DLL Boundaries

2009-06-24T09:31:00.000-07:00

Resource management across DLL boundaries in C++ is tricky. Here are the basic rules used in Boost.Extension:

Anything allocated inside the DLL must be deallocated before the DLL is released. This means that if you pass a std::map into the DLL and modify it, you must delete (or at least clear()) the std::map before releasing the DLL.

In order to use an interface across the DLL boundary, you basically have to guarantee that both the DLL and main executable were compiled with the same options - this is, for me, the primary weakness of C++ with DLLs.

RTTI does not always function as expected across DLL boundaries. Check out the type_info classes to see how I deal with that.

Always use virtual destructors for your interfaces - then you can safely delete them from the main binary or in the DLL.

It took a whole lot of trial and error to get this to work on different platforms.

Translation Bar

2008-08-01T10:27:00.000-07:00

At least 50% of the visitors to this web site do not speak English as a first language. I have placed a Google Translation box on the left. Hopefully this will help make it easier to understand.

However, I realize that the translations it gives aren't very good - especially when it tries to translate source code. Don't misunderstand me - the Google software is quite impressive, but this is the type of text where a native speaker (who is also a programmer) is needed for a good translation.

If you would like to provide translations of any of the documentation content, I will post it here. The most important parts are the first two tutorials.

Download Options

2008-08-01T10:03:00.000-07:00

There are a few different ways to install Boost.Extension and Boost.Reflection.

I've provided a download, to the left here, that includes both Boost.Extension, Boost.Reflection, and all Boost libraries needed for the examples and tests.

But, if you have Boost installed, you don't need all the other libraries, so feel free to just copy the boost/extension or boost/reflection directories to your Boost tree instead.

If you want to always stay up to date with the latest changes, you can also get the libraries directly from the Boost Sandbox using Subversion:

svn co http://svn.boost.org/svn/boost/sandbox/boost/reflection
svn co http://svn.boost.org/svn/boost/sandbox/boost/extension

Let me know if you have problems installing or compiling on any system.

Also, as a note, the libraries themselves are header-only.

Final Changes Before Review

2008-07-26T23:56:00.000-07:00

Since I presented these libraries at BoostCon in May, I've been working to address a few of the feature requests mentioned there, and clean up the library for review for inclusion in Boost.

I could use your feedback on the following:

1 - What do you think is missing in the libraries?
2 - Have you had any problems getting them to work? Which OS? Compiler?
3 - I've updated the first three tutorials quite recently. Are they clear enough?
4 - I've updated the header references recently (with Doxygen). Are those docs clear enough?
5 - Anything else you can think of.

Feel free to post a comment to this blog, or to send me an e-mail at rostovpack [at] gmail [dot] com.

Just as a heads up, here are the things I plan to finish before review:

The library headers themselves:

1 - Add an optimization for the case where factories or reflected functions with unknown parameters are called.
2 - Move the entirety of the Boost.Reflection library to the boost::extensions namespace and the boost/extension directory (because of how much the libraries overlap, I'd like to have them together for review).

The documentation:

1 - About 10 total tutorials - 3 are up to date, and one more is just about finished. The remaining tutorials will cover calling functions without knowing their signatures, compiling and loading new shared libraries at run-time (ie, c++ interpretation), reference-counting within shared libraries, storing version information etc. with factories, and some related topics.
2 - Rewrite performance analysis section to take into account recent changes.
3 - Expand FAQ and appendices.
4 - Add installation instructions.

The examples:

1 - Examples for each of the tutorials listed.
2 - An example plugin-based image processing system.
3 - An example using GCCXML to automatically generate factory or reflection declarations.

The build files:

1 - I need to clean up and simplify the Jamfiles. I need to use some of the less well-documented features of Boost.Build to do it.

Once the above are done, I plan to submit the library for review.

While reading the docs, the main page lists at the bottom all of the parts that have been updated recently. Those are the most useful parts to read.

If there's anything else that you want to see, please let me know.

The redistributable is on the left in tar.gz format.

Latest Release - Downloadable

2008-07-15T15:59:00.000-07:00

I've packaged together the latest stable release:

extension_reflection.tar.gz

It contains the tests, examples, documentation and headers. It also contains the other Boost libraries that these depend on. Most of the other libraries are used by the examples, but not required for using the headers.

To use the library in your project, just add the root of the extracted directory to your include path (ie, /home/me/downloads/extension-reflection).

If you want to make sure that the tests pass on your system, you will need bjam - see boost.org. Then run the test_redist.sh script. If bjam is not in your path, you'll need to set the variable BOOST_BJAM_PATH. You can also set BOOST_BJAM_OPTIONS if needed.

I'll provide a Windows .bat file in the next release.

For the documentation, I've only updated the Extension reference section. The docs are at extension-reflection/libs/extension/doc/html/index.html in the redistributable. I'll post the latest docs up on this site once I've fixed some of the tutorials.

Feel free to let me know if you have trouble getting the redistributable to work on a given system - it is a work in progress.

Also, it can take a little bit of setup to use Extension without Bjam. If you've worked it out for your compiler/OS, feel free to send me your instructions and I'll try to post them with the documentation.

Back Up!

2008-04-09T18:41:00.000-07:00

All data is back up. URLs should be the same as they were previously.

Sorry for any inconvenience!

Edit: Oops, some of my links were bad. This is fixed now.

Temporary Server Outage

2008-03-26T10:08:00.000-07:00

I'm moving the documentation to a different server. As such, you'll need to get it by downloading the library directories from SVN (see earlier posts). It should all be there.

Sorry for the inconvenience!

Downloading the Libraries

2008-02-12T19:17:00.001-08:00

As I've mentioned before, the easiest way to get these libraries and stay up to date, is to get them from the Boost Sandbox. However, the latest version of the libraries is also available here:

Extension/Reflection tar.gz file

This archive also contains each of the Boost libraries used by the Extension or Reflection library, or by the examples.

Reflection/Extension interoperability

2008-02-09T20:25:00.000-08:00

I've posted one single page of the Reflection docs - an example of reflecting classes that are located in shared libraries, using the Extension library. Since this is perhaps the most common use case for the library, I figured I'd post it even before the rest of the docs were done:

Boost.Extension/Boost.Reflection interoperability

Docs Posted for Boost.Extension

2008-02-08T23:28:00.000-08:00

I've posted the docs for Boost.Extension at:

Boost.Extension Documentation

Almost half of the doc material was written by Mariano, the rest by myself.

Boost.Reflection docs will be posted as soon as they are completed.

Back at Work

2008-02-08T22:15:00.000-08:00

My apologies for the long hiatus!

Currently, both libraries compile with:

MSVC 2005
GCC 4 (Ubuntu)
GCC 4 (OS X)

If you have any questions about compilation or installation, please let me know (just in the comments of this post is fine, or e-mail me at rostovpack - at - gmail.com.

The Reflection documentation is incorrect - my next project. For examples of using reflection, I recommend looking the libs/reflection/test/ directory. The latest test is parameter_info_test.cpp, which allows you to store information about the arguments in a reflected function (for instance, names of the parameters). I haven't yet written in the code to actually retrieve this information, but it should be finished soon.

If you see any compilation errors or warnings, please let me know.

Current work:

1 - Integrate the new Boost.Function/Factory functionality into Boost.Extension.
2 - Update Reflection documentation and examples.
3 - Uses for the parameter_info.

Any feature requests?

What is Boost.Reflection?

2007-09-06T10:11:00.000-07:00

Boost.Reflection is similar to, and can be used together with, Boost.Extension. However, it allows for creating and using objects without having access to the definition of the class or one of its base classes (Boost.Extension requires having access to one of a class's base classes).

Features:

Use arbitrary information to describe a class
Instantiate a class
Call its methods by arbitrary identifiers (although you can use strings to identify methods, it is possible to use any arbitrary information to select them)
Fast function calls - you only have the overhead of two functions called by function pointers.
Call functions without knowing what parameters they require.
Containers indexed by type.

Information about Boost.Reflection is currently contained in the Extension docs. Especially check out the following:

reflection.hpp reference
reflector reference
Extension/Reflection interoperability

What is Boost.Extension?

2007-09-06T10:04:00.000-07:00

Boost.Extension is a C++ library designed primarily to ease the creation of plugins. It allows for creating shared libraries that can be loaded by an executable, and which can implement interfaces defined in the executable. The executable can then instantiate and use these implementations.

Features:

Type-safety
Arbitrary information can be stored about a plugin (version info, size, requirements etc.)
Fast - you access implementations through pointers to their interfaces, so you only have the normal overhead of a virtual function call
Thread-safe
Cross-platform (Windows, Mac, and Linux have all been tested)
RTTI is optional

Status

2007-09-04T10:11:00.000-07:00

I've revamped the interface for Boost.Extension, to make it more concise. The main difference is that the RTTI requirement is easier to avoid now.

Option 1:

You can have separate containers for each set of constructors for a specific interface.

Option 2:

You can declare your own type of type info - perhaps just integers or strings.

The latest change makes option one easier.

A recent checkin in Boost.Build seems to have broken the build for documentation. If you have trouble generating the documentation, you may have to wait a few days until I check the latest docs into svn.boost.org in the sandbox. I managed to get the docs compiling successfully on Ubuntu, and should have the updates posted in three to four days.

2007-08-30T14:06:00.000-07:00

I finally have the redesigned functionality for reflection. It's not done certainly, but the following will compile and work correctly. I'll be committing this to the SVN repository soon.

Here's how it works:


/////////////////////////////////////
//  In your code that reflects some class out:

class MyClass {
public:
  int myFunction();
}

// For terseness
using namespace boost::reflections;
generic_reflector * exportMyClass() {
  reflector * myClassReflector =
  new reflector;
  // reflect an arg-less constructor
  myClassReflector->reflect_constructor();
  //  reflect the member function
  //  with some info about it
  //  note that like boost::extensions,
  //  the info parameter is templated
  //  with a string as the default
  myClassReflector->reflect_method(&MyClass::myFunction, "myFunction");
  return myClassReflector;
}


///////////////////////////////
//  Now in the code that will use the reflection
//  (This code could be in a DLL, or the top code could be in a DLL etc)
using namespace boost::reflections;
  int main(int argc, char * argv[]) {
  //  A reflection must be created with a pointer
  //  to a generic_reflector
  //  The reflection will destroy the generic_reflector
  reflection myReflection(exportMyClass());
  //  Create an instance with a default constructor
  instance inst = myReflection.get_constructor().call();
  //  Find the function with an int return value and
  //  info of "myFunction" and call it.
  function func =
       myReflection.get_method("myFunction");
  //  Call func on inst
  func(inst);
}



////////

2007-08-23T09:50:00.000-07:00

One conversation from the Boost Mailing list contained a lot of the rationale behind both libraries. The basic design goals were the following, in order of importance:

Cross-platform (this is Boost - it has to be)
Concise, natural interface (no macros, etc.)
Minimization of dependencies on other Boost libraries
Separation of optional features of the library from the primary headers
Performance

Number 3 was difficult, and we've decided to probably require inclusion of Boost.Preprocessor. Constructors or other functions with many arguments require a lot of template code that is much easier to just generate using Boost.Preprocessor. Boost.Function will probably be required as well.

For some of the original discussion on some of these design decisions, see the discussion that began with Mariano's interest in writing a plugin library for the Google Summer of Code:

Current Progress

2007-08-23T09:28:00.002-07:00

Mariano Consoni has been working on both Boost.Extension and Boost.Reflection over the summer. Our current status on these two projects:

1 - Boost.Extension is mostly complete. One piece of functionality that I have not finished is the ability to automatically close shared libraries that do not currently have any loaded plugins. The main obstacles here are preserving the current Boost.Extension syntax and overcoming some platform irregularities (on Mac OS X there are two types of dynamically linked library with very different properties - bundles and dylibs).

2 - I am currently working with the code that Mariano wrote for Boost.Reflection to make it a bit more generic and do some cleanup and performance tuning.