Running a single CFD simulation rarely provides enough information to find a good hydraulic or aerodynamic design of a component. In many cases the poor engineer has to wrestle an awfully long list of design parameters into a state-of-the-art shape. Luckily, OpenFOAM and DAKOTA together provide an almost ready-to-use solution just for that:
- Sensitivity analysis
(long story short: checking which variable has the greatest influence on the result)
(finding the optimum combinations of inputs for the best output)
- Model (curve, surface, …) fitting
- Uncertainty quantification
(I have no idea what that is)
- And more
(I also have no idea about all that)
This post is a guide to OpenFOAM and DAKOTA from installation to an optimized solution.
A Real-Life Solution
I’m building a wind turbine and right at the beginning my party was pooped by the most important element – the blades. I decided to cut them from pipes and scripted a program (called model from now on) that takes geometry parameters, runs a simulation and returns rotor torque. It’s rather complicated-ish so I dedicated a separate blog post to it. Have a look.
Once you have a model from which you can obtain results it’s just a matter of wrestling them into a combination that works best. Here, DAKOTA comes to help with its optimization talents. The recipe for working with it looks like this:
- Specify input parameters for your model: ranges, values, names and whatnot.
- Specify the model’s output – value we’re trying to minimize or maximize.
- Point DAKOTA to your model (a script that does the calculation).
- Choose which algorithm to use.
- Run and wait. A long long time.
- Check the results. The winning combination should be somewhere in the output files.
OpenFOAM and DAKOTA Installation
If you are reading this, you most probably already have OpenFOAM installed on your favorite platform. I’ll write instructions for installation of software I use because it’s not completely obvious and/or quirk-free. Even if no one needs that, I’ll have this guide as a note to self.
Windows: WSL2 and Ubuntu 20.04
If you’re using Ubuntu or any other Linux distribution, you don’t need this. But if you’re on Windows, installation procedures are well-documented: https://docs.microsoft.com/en-us/windows/wsl/install-win10
I really recommend WSL2 over the original WSL. The latter had some issues with disk I/O (my explanation) which caused apps to hang forever at totally random occasions. With WSL2, this problem seems to be gone.
I also recommend installing the latest Ubuntu version to WSL, that is 20.04 LTS at the time of writing. What’s great about it is that python3 is now default and python2 is nowhere to be seen. This really helps to reduce the confusion. (Not that it’s that important, though.) (Also not that it’s the only good thing about the latest Ubuntu.) (I actually don’t know what’s the difference between any of the versions.)
If you need to calculate pretty much anything, you’ll need
scipy. Those two packages alone are quite heavy but you’ll probably need a few more. To keep them separated from other python things you and your system need, I highly recommend putting your calculation stuff in a virtual environment. I wouldn’t if it wasn’t so simple. For instance, I keep my wind-turbine-related environment in
sudo apt-get install python3-pip sudo apt-get install python3-venv python3 -m venv ~/venv/turbine
The environment has been created. To work in it, activate it with:
You have to do this every time you open a new shell. If you want to deactive the environment to work on another one, just type
Now, install whatever you need to do your magic:
pip3 install numpy scipy jinja2
With this setup, you should be ready to run a single iteration from this post.
The guys at Sandia are a little stingy with binaries – they only provide one for Linux, and that is for Red Had Enterprise, which is something I do not have. So you have to compile it from source. It takes a while but with these instructions hopefully you’ll make it.
First, prerequisites: stuff you need for compilation.
sudo apt-get install gcc g++ gfortran cmake libboost1.67-all-dev libblas-dev liblapack-dev libopenmpi-dev openmpi-bin gsl-bin libgsl-dev python perl libhdf5-dev cowsay
Then, create a directory to hold source and binaries. To keep things simple, chown it:
cd /opt/ sudo mkdir dakota sudo chown $USER dakota cd dakota
You’ll also need the source. Download it from this page. If it’s a bit confusing – select Source (Unix/OS X), then Release 6.12, Command Line Only, Supported. When download finishes, move it to this directory and unzip it:
cp /mnt/c/Users/<user>/Downloads/dakota-6.12-release-public.src.tar.gz . tar -xvzf dakota-6.12-release-public.src.tar.gz rm dakota-6.12-release-public.src.tar.gz
Now you’re ready for compilation. Make a build directory, configure and compile. We’ll stick to defaults so there’s no need for all complications in the official instructions.
Don’t forget to change the number of processors for make command. For a bonus little stress-test of your computer.
mkdir dakota-build cd dakota-build cmake ../dakota-6.12.0.src/ make -j <1.5x number of your physical cores> # This might take a while... Stay strong! make install .
Hopefully everything went OK. If it did, you can run DAKOTA:
cd ~ nejc@Buquica:~$ dakota --version Dakota version 6.12 released May 15 2020. Repository revision 35a9d0d5b (2020-05-13) built Oct 15 2020 22:12:47.
Installation and Compilation Caveats
libboost-all-dev breaks something with version 1.71 so it throws an error
boost_find_component Macro invoked with incorrect arguments for macro. The newest previous available version for Ubuntu 20 at the time is 1.67 – so we install that.
The last step of OpenFOAM installation requires you to add a
source command to
source /opt/OpenFOAM/OpenFOAM-v2006/etc/bashrc). This now forces other programs to use OpenFOAM’s libraries which makes them very miserable (
dakota is among them):
dakota: /opt/OpenFOAM/ThirdParty-v2006/platforms/linux64/gcc-6.3.0/lib64/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by /usr/local/bin/../lib/libteuchosparser.so.12)
I did not go too deep into research why this happens – all my problems were solved by adding another line to
Restart the shell and both OpenFOAM and DAKOTA should now coexist peacefully:
simpleFoam --help dakota --version
OpenFOAM and DAKOTA: Workflow
Just to remind you: we’re shaping a wind turbine blade cut from a pipe. We’re searching for the best combination of parameters – the one that will give the highest torque output.
This is a black-box script from another post. It takes four parameters: inlet angle and (angular) blade span at root and tip of the blade. Other four parameters are fixed – I chose turbine diameter and available pipe dimensions in advance. I just want to know how to cut it.
___________________________________ / This design gives you a torque of \ \ 0.005918680e-03 Mooton-meters / ----------------------------------- \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || ||
We’ll be dealing with a single chapter among DAKOTA’s capabilities: optimization. There’s no point in explaining this stuff because it’s already much-too-well explained elsewhere. Our algorithm must meet a few criteria:
- multiple input parameters
(4 in this case)
- no gradients
(the simulation can calculate a single point and that’s it)
- noisy data
(resulting function will not be microscopically smooth)
- concurrent evaluations
(that isn’t critical but it would be nice to work on multiple points at once)
Almost anything can manage #1 and #2. But! #3 and #4 are bad news. Anyhow, some can do all of the above so I chose the dividing rectangles method. It proved quick and robust not only in this case but also in optimization sessions with 7 variables and more.
Before I bury you with tons of configuration text, let’s sum up what I want OpenFOAM and DAKOTA to do for me:
- Find a combination of 4 blade parameters
- that give the maximum torque,
- using a dividing rectangles method,
- stopping at a certain tolerance,
- limiting input values to physically manufacturable shapes.
- Provide additional 4 parameters
- that don’t change between iterations.
- Obtain blade torque from each iteration
- by running my custom black-box model
- which cannot provide information about gradients or hessians or whatever.
- Write all results to a text file.
That’s it. All that is left to do is to translate the above list to whatever DAKOTA will understand. The
blade.in file explains itself:
environment # write results to this file tabular_data tabular_data_file = 'results.dat' method # use the dividing rectangles method; coliny_direct # and don't optimize beyond precision # our model can provide min_boxsize_limit = 0.2 variables # specify our 'design' variables - parameters active design continuous_design = 4 descriptors 'beta_inner' 'arc_inner' 'beta_outer' 'arc_outer' lower_bounds -180 30 -180 20 upper_bounds 180 175 180 95 # initial point is optional for dividing rectangles # (see chapter mapFields) initial_point -90 90 -90 90 # these parameters will be included in the input file but will remain # fixed with these values continuous_state = 4 descriptors 'r_inner' 'r_outer' 'pipe_diameter' 'pipe_thickness' initial_state 0.12 0.75 0.125 0.004 interface fork # dakota will 'fork' this script for each iteration; # two command-line parameters will be appended, # input file path (iteration parameters) # and output file path (results) analysis_driver = 'run' responses # at the end of each iteration, DAKOTA will # read the output file and look for a variable named 'torque' objective_functions = 1 descriptors 'torque' # what our model can't provide no_gradients no_hessians # we're not minimizing but maximizing # our function sense 'max'
Parsing and Passing Parameters
By default dakota will create an input file somewhere in
/tmp/ directory and pass its path as the first argument to our
run script. The file will contain all variables that we need but in a totally useless format. Fortunately guys at Sandia figured that out themselves and provided a preprocessing script that creates a more pleasant format. We can write a template file where we specify the input format. If we write it correctly, we can store the contents of the file directly into variables – which are then available throughout the
run script as
dprepro $parameters_file parameters.template parameters.in # the parameters.template is written in a way it can be read directly: . parameters.in
At this point our scripts are ready to run but there’s one more detail I’d like to clarify before your computer catches fire.
Since we will be running dozens of iterations with very similar geometry, we can speed up the convergence by initializing our fields from previous iterations.
The first iteration will start un-initialized and wil therefore need a little longer to converge. It will save the results to a separate case which will then be used by all other iterations. They will start from a better initial
internalField and converge much, much quicker.
The first iteration will know it’s the first by reading
results.dat file that dakota writes to live.
iteration_no=`wc -l < results.dat` if [[ $iteration_no == "1" ]]; then # first iteration: initialize fields with potentialFoam runParallel potentialFoam else # all next iterations: map fields from 'initial' runApplication mapFields -consistent -parallelSource -parallelTarget -mapMethod mapNearest -sourceTime latestTime ../initialized fi #runParallel simpleFoam #...
After the first iteration is done, it saves the
case directory to
initialized for the successors.
cd .. if [[ $iteration_no == "1" ]]; then rm -r case_zero cp -r "case" case_zero fi
Now we call
dakota and provide the above file as input with an
-i blade.in option. If you don’t want it to throw the output text up all over your terminal, provide an
-o output.log option:
dakota -i blade.in -o output.log
Now wait for the results. A long long time.
If everything went OK,
results.dat file we specified in blade.in will contain parameters and outputs from all iterations. At the end of
output.log you can find which iteration was the best. You’ll have all the parameters if you open the results.dat file and look for the mentioned iteration.
%eval_id interface beta_inner arc_inner beta_outer arc_outer r_inner r_outer pipe_diameter pipe_thickness torque 1 NO_ID 0 102.5 0 57.5 0.12 0.75 0.125 0.004 -0.0832453759 2 NO_ID 120 102.5 0 57.5 0.12 0.75 0.125 0.004 -0.0842540585 3 NO_ID -120 102.5 0 57.5 0.12 0.75 0.125 0.004 0.0175737057 ... ... ...
Now it’s time to stop crunching numbers and start drawing!
A Few More Hints
If you have a sick number of cores available or if your domain is stupidly small (read: 2D), instead of over-decomposing it might be a better idea to run multiple iterations at once, each with a single core (or very little of them).
evaluation_concurrency is the keyword here. Keep in mind that how many you can run is also determined by the algorithm and number of free variables. Here’s what the docs say about dividing rectangles: “The DIRECT algorithm supports concurrency up to twice the number of variables being optimized.“
In this case you will need to isolate iterations by putting them in separate directories. See the description of
work_directory keyword. Most probably you’ll also need some config files/scripts to work with: see
Keeping Iteration Results
Input and Output Filter
In this tutorial we wrote a single script that handles everything. Geometry calculations, model generation, simulation set-up and postprocessing. For a more complex model it might be convenient to have separate scripts. For instance, a python script for calculations and configuration, an Allrun shell script and another python script for postprocessing calculations. That’s why dakota offers
output_filter keywords. Even more, you can specify multiple scripts to be run for every command.
Reset and Failure
If your optimization session is stopped for any reason you can continue from the last successful step. By default DAKOTA creates a dakota.rst file. If
dakota -i blade.in -read_restart dakota.rst it will try to continue where it left off.
When an iteration crashes/fails/doesn’t return a result, DAKOTA will lose it and refuse to continue. Here’s how you handle that:
- Decide how to handle failures: for my cases,
recoverwas the best option.
- Decide which number to use on fail. For instance, when optimizing efficiency, a 0 might be in order. When looking for a greatest torque, a sufficiently negative number will do.
- The first thing your scripts do should is to write a results file. The contents:
- If the iteration succeeds, rewrite the results file with… results.
- If the iteration fails, the fail string will tell DAKOTA something went wrong and it wil use the number from #3 as result.
DAKOTA can optimize functions that return many values. It does that by combining outputs with weights the user specifies. I have no experience with multiobjective optimization. Anyway, you might still want your model to return multiple values (for instance, lift and drag, weight, efficiency, whatever). In this case you can specify multiple
response_functions. Then you assign
weights of 0 to every response except the one you’re optimizing.
Don’t forget to update
recover with a separate value for each response.
A More Proper Configuration
Here are all of the above hints translated to DAKOTAian. Just so you know where to put all the stuff.
interface asynchronous evaluation_concurrency = 8 fork analysis_driver = 'Allrun' input_filter = '/opt/blade/prepare.py' output_filter = '/opt/blade/postprocess.py' work_directory copy_files '/opt/blade/case' named 'dakota_work/iteration' directory_tag directory_save file_tag file_save failure_capture recover -100 0 0 100 responses objective_functions = 3 descriptors 'torque' 'lift' 'drag' 'mass' weights 1 0 0 0 no_hessians no_gradients sense 'max'
OpenFOAM and DAKOTA Files
Here’s the whole blade optimization thing without results. It should work on a setup described above.
Have fun and good luck!