An Interface for Watershed Evaluating of Agricultural BMPs Using SWAT, C#, DotSpatial and SQLite

Copyright: I have got the permission from project partners to publish these information. The copyrights of the data and related software are hold by all project partners. All the information here is for demonstration purpose only. It can’t be used for any other purpose. For any concerns, please don’t hesitate to contact me.

I have been working on various projects and have developed several interesting products. I guess they could be part of my “portfolio”. From today, I would try to summarize some of them and post here. The very first one is an interface developed in 2012 when I was a post-doctoral in University of Guelph, Ontario, Canada.

Click here to download a PPT for this project

OVERVIEW

When: 2012

Where: South Tobacco Creek watershed near Miami, MB

Team: Watershed Evaluation Group, University of Guelph

Project: Watershed Evaluation of Beneficial Management Practices (WEBs), AAFC

Objective: Develop an interface to create and evaluate what-if agricultural BMP scenarios to help conservation districts/authorities gain understanding on water quantity and quality effects and allocate limited resources for BMP implementation

BACKGROUND

Agricultural best management practices (BMPs) has been applied in Canada to help reduce nitrogen and phosphorus coming from farm fields. With limited budget and clear water quality goal, conservation districts/authorities need to choose the most cost-effective BMPs and their locations. Managers needs a decision support system (DSS) to help them understand the impact of various available BMPs on hydrology and economy, i.e. what-if scenarios, and then make proper decisions.

The Watershed Evaluation of Beneficial Management Practices (WEBs) program was a successful nine-year Government of Canada initiative to determine the economic and water quality impacts of selected agricultural beneficial management practices (BMPs) at nine watershed sites across Canada. The interface was developed as part of the program for the South Tobacco Creek watershed located near Miami, Manitoba.

FUNCTIONS

1. Create scenarios to add, remove and configure various BMPs
2. Run SWAT and economic model for each scenario
3. Display SWAT, economic and integrated results on field, farm, subbasin and watershed level with map and chart

TECHNOLOGIES USED

The interface is a Windows form application developed using C#. The main part of the interface is the map area implemented with DotSpatial map control. Microsoft chart controls are used for all charts.

In model side, to enable fast access to SWAT results, SQLite is added to SWAT source codes to write all results into a SQLite database with fixed structure including tables, views and indexes. A scenario definition file is also generated as the bridge between the interface and SWAT model to exchange data. Mix-language compilation (C and Fortran) is required to compile the modified SWAT codes.

The system architecture is show below.

ROLES AND RESPONSIBILITIES

1. Design the main architecture and choose required technologies
2. Design the GUI and code with C#
3. Design the interface between GUI and SWAT with the scenario definition file
4. Modify SWAT sourced codes to support SQLite results and scenario definition file

SCREENSHOTS

Project tree on the left to navigate different views, including project, scenario, BMPs and results

Project View

A project represents a watershed. It includes the geospatial and model information.

Scenario View

A scenario is a combination of various BMPs including small dams, holding ponds, grazing, tillage and forage conversion. SWAT and economic model could be run for each scenario. Results could be viewed after the simulation is done.

Small Dam View

Select/deselect small dam on map or from the list on the right to enable/disable them in model

Tillage View, Field Level

Select/deselect fields on map or from the list on the right to apply tillage. Farm level and subbasin level is also supported.

Result View, SWAT Results

• Display results with different colors for polygon and different sizes for point
• Select a feature to view time series in chart at the bottom
• Control result display using the right panel

Result View, Economic Results

Result View, Integrated Results

• Cost-effectiveness, i.e. the amount of reduction (on water, sediment, N or P) per $1000 between two scenarios.
• Most valuable for conservation districts and authorities

Result View, Off-site Integrated Results

Off-side is for the result at the watershed outlet and is good for whole watershed evaluation.

Advertisement

Work Flow of Point Source and Reservoir Monthly/Daily Discharge File Generation

When reservoir outflow is simulated with measured daily/monthly outflow, a discharge file (.day/.mon) would be generated. The file would be overwritten when the simulation is setup with the given simulation period. For those who don’t use ArcSWAT/SWAT Editor to run the model, remember to setup simulation before running the model if there are reservoir or point source in your watershed and the daily/monthly discharge method is used.

The data flow of reservoir discharge data in ArcSWAT/SWAT Editor is described below. Point source discharge data (.DAT) also follows exactly the same data flow.

1. Prepare the original discharge data in dbf/txt format

Daily

“DATE”, “RESOUTFLOW”

1/1/1978,0.00

1/2/1978,0.00

Monthly

“Year”,”Resout1″,”Resout2″,”Resout3″,”Resout4″,”Resout5″,”Resout6″,”Resout7″,”Resout8″,”Resout9″,”Resout10″,”Resout11″,”Resout12″

1990,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0

1991,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0

Note: The comma in the column name is a must. You may expect an error message shown below if they are missing.

2. Import the discharge data in ArcSWAT/SWAT Editor. The discharge data would be imported to timeseries data in project mdb. Note that TSTypeID is 0 and the time is same as the original discharge file. The data here should cover the future simulation period.

3. Re-write .Res/.Lwq, which would re-write all RES file and possible MON/DAY file. Note that the data in the generated DAY file doesn’t started from Jan 1, 1990 (in which day the discharge is 0.07 m³/s) as the simulation period hasn’t been set. ArcSWAT/SWAT Editor may set an arbitrary starting and ending date here to make the time range big enough (like from 1/1/1000 to 1/1/3001). This is not the final version used in simulation.

.DAY

Daily Reservoir Outflow file: .day file Subbasin:40 9/25/2014 12:00:00 AM ArcSWAT 2012.10_1.15

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

0.00

4. Setup Simulation, the MON/DAY file would be overwritten again to extract discharge data fell between the starting date and ending date from timeseries datatable. Now the data in DAY file starts from Jan 1, 1990.

.DAY

Daily Reservoir Outflow file: .day file Subbasin:40 9/25/2014 12:00:00 AM ArcSWAT 2012.10_1.15

0.07

0.07

0.07

0.07

0.07

0.07

0.07

0.07

0.07

Default CN2 in MGT

A default CN2 is defined in mgt file as shown in following examples.

49.00 | CN2: Initial SCS CN II value

Question 1: Where does this default CN2 come from?

The default CN2 comes from either crop table or urban table of SWAT2012.mdb. Four columns (CN2A, CN2B, CN2C, CN2D) are defined in table crop/urban, which are CN2 values for soils with hydrological group A, B, C and D respectively. HRU is a combination of unique landuse, soil and slope. From the soil type, the hydrological group is obtained from usersoil table. And then depending on landuse type, the CN2 value is read from table crop or urban.

Urban is quite unique compared to crop. The default CN2 is just for pervious surface. For impervious surface in urban area, URBCN in table urban would be used. Its value is usually 98.

Question 2: What’s the impact of the default CN2 on model results?

The CN2 in mgt file is the initial CN2 for the HRU. If the CNOP is NOT defined in plant/harvest/tillage operations, the curve number used in infiltration calculation would just depends on the default CN2 and soil moisture. In this case, the default CN2 would have big impact on infiltration and surface runoff and further on flow discharge. That’s why this is usually the main calibrated parameter.

However, the default CN2 could be changed to CNOP if it’s defined in any plant/harvest/tillage operation, where the default CN2 would be only the initial value and would only have impact before it’s changed. In this case, it’s the CNOP we should calibrate rather than CN2.

So, for some scenarios which focus on tillage or crop change, it’s very important to set CNOP.

Question 3: What if the default CN2 is 0?

SWAT model would check the default CN2 and make sure its value is between 35 and 98. See following codes from readmgt.f.

if (cn2(ihru) <= 35.0) cn2(ihru) = 35.0

if (cn2(ihru) >= 98.0) cn2(ihru) = 98.0

 

SWAT Changes from Rev 622 to Rev 627

Updates July 17,2014: The SWAT Team had confirmed that prf_bsn should be real rather than integer. It would probably be fixed in next release.

SWAT is updated on June 24, 2014 to Rev 627. It gives modelers more control on three parameters (prf, r2adj and surlag) and adds more outputs for auto-irrigation operation in output.mgt. Let’s see the details.

Changes from SWAT Version History

Revision 627

• ‘surlag’ input (.bsn) changed to ‘surlag_bsn’ (if the input for surlag is <= 0 in the .hru file, the model will use surlag_bsn from .bsn file – defaulted to 4.)
• ‘prf’ taken out (.hru) and changed to ‘surlag’ (if the input for prf is <= 0 in the .hru file, the model will use prf_bsn from the .bsn file defaulted to 1.)
• Autoirr issues in output.mgt resolved (reporting issue)
  • NOTES regarding the autoirr changes: The SCHEDULED irrigation from the .mgt input file will be labelled as: “SCHED AUTOIRR” in the output.mgt file. The actual applications will be labeled in same file as “AUTOIRR”. The ‘scheduled irrigation’ is when it was scheduled in the .mgt rotation. AUTOIRR is when it actually was triggered and applied.
• Subdaily problem with reservoirs fixed

Revision 626

• No significant changes.

Revision 625

• ‘r2adj’ input (.bsn) changed to ‘r2adjbsn’
• ‘prf’ input (.bsn) changed to ‘prf_bsn’

Revision 624

• Minor subdaily changes
• Format extended for MSK_K in input.std output file (from f6.2 to f8.2)

Revision 623

• No significant changes.

 

More Details from Analysis of Source Codes

1. prf
   

Description

Reach peak rate adjustment factor for sediment routing in the channel. Allows impact of peak flow rate on sediment routing and channel reshaping to be taken into account.

Modification

In previous version, it’s a basin level parameter defined in .bsn file and each reach share the same parameter value.

Now, it’s a reach level parameter. Each reach could define its own value. It’s added to end of .rte file and the default value is 0, in which case the value given in .bsn value will be used (prf_bsn).

Potential Problem

In previous version, the variable for prj read from .bsn is defined as real. In new version, it’s defined as integer. When trying to run new version on model generated from a previous version of ArcSWAT, it may give following errors for trying to read integer value from a real value. Changing the real value to integer (e.g 1.0000 to 1) would solve this problem.

At line 386 of file readbsn.f (unit = 103, file = ‘basins.bsn’)

Fortran runtime error: Bad integer for item 1 in list input

Only gfortran compiled SWAT executables would have this problem. The Intel Fortran compiled version would work without problem. Should this variable be defined as real?

2. r2adj
   

Description

Soil retention parameter adjustment factor (greater than 1)

Modification

Similar as prj, the basin level parameter is refined to HRU level. You would more control on this parameter for different HRU. It would be read from HRU file after surlag and default value is 1. The value defined in .bsn file (r2adj_bsn) would be used when the HRU value is less than or equal to 0.

3. surlag
   

Description

Surface runoff lag time. This parameter is needed in subbasins where the time of concentration is greater than 1 day. SURLAG is used to create a “storage” for surface runoff to allow the runoff to take longer than 1 day to reach the subbasin outlet.

Modification

Similar as r2adj, this basin level parameter is refined to a HRU level parameter. It would be read after n_lnco and before r2adj. If it couldn’t be read, it will be set as surlag_bsn defined in .bsn whose default value is 4.

4. Auto Irrigation
   

Add output information for irr_rch.f and irr_res.f to output.mgt. It’s still called “AUTOIRR” rather than “SCHED AUTOIRR” described in SWAT version history.

 

 

 

 

Single Document about Compiling and Debugging SWAT with GFortran and Eclipse

After several posts about compiling and debugging SWAT with GFortran and Eclipse, I think it would be good to compile all the posts into an single document as a guide on this topic. So I create a google doc here and would like invite all of you to give some comments.

Makefile Updated – 1) No Need to Modify main.f any more; 2) A Single Makefile; 3) 64-bit Build

Download Link

Makefile Generator, Makefile

The Makefile is updated again. Main changes are:

1. Don’t compile modparm.f any more to keep SWAT codes unchange.
2. Don’t generate separated Makefiles for debug and release any more. A single Makefile is generated instead to include four make targets: debug32, debug64, rel32 and rel64, which matches the four official SWAT executables.
3. 64-bit build added to the Makefile.

No Need to Modify main.f any more

In previous version of Makefile, to compile SWAT successfully, the first line of main.f (include ‘modparm.f’) must be comment/deleted, which is not so convenient but not so bad.

It has been found that the problem is in the compilation of modparm.f and main.f. The parm.mod would be generated when either modparm.f or main.f is compiled. The key here is the effect of include statement., which could be found in gnu website.

The effect of the INCLUDE directive is as if the included text directly replaced the directive in the source file prior to interpretation of the program.

So, include ‘modparm.f’ means adding all the contents of modparm.f to main.f, i.e. all the variables are also defined in main.f. If both of modparm.f and main.f are compiled to object files and used to in the link operation, all the variables would be defined twice and the linker would give error message shown below.

Solution is just very simple: don’t compile modparm.f at all. Thus, all the variable would be just defined in main.f and include ‘modparm.f’ doesn’t need to be comment/deleted.

A Single Makefile

The new Makefile would be put in the source codes folder rather than the debug and release folder. Sub-folders (debug32, debug64, rel32 and rel64) would be created to hold all the object files and mod file. The final executable files will be in the source codes folder and given different name similar as official SWAT executables. More details are list in the following table.

Command	32-bit or 64-bit	debug or release	executable name	Comment
make debug32	32	debug	swat_debug32
make debug64	64	debug	swat_debug64	Doesn’t support by MinGW
make rel32	32	release	swat_rel32
make rel64	64	release	swat_rel64	Doesn’t support by MinGW

64-bit Build

64-bit build is added to the Makefile though flag -m64, although MinGW doesn’t support it. To compile 6-bit SWAT, either use MinGW-w64 or Rtools instead. More details would be published shortly in an integrated document.

SWAT Makefile Updated – Ignore Underflow

The SWAT Makefile is updated again. Underflow flag is removed to get same results as official SWAT.

Download Link

Makefile Generator, Debug Makefile, Release Makefile

In an attempt to run SWAT (rev 615) in Eclipse and compare with official SWAT, although official SWAT could run to end without problems, Eclipse-SWAT stops running due to underflow (erroneous arithmetic operation).

The underflow happens in subwq.f Line 88.

wtmp = 5.0 + 0.75 * tmpav(j)

where

tmpav is average air temperature on current day in HRU

From the Expression window, the value of tmpav(j) is 4.6281851e-038. The underflow should happen when 0.75 * tmpav(j) is calculated.

The fact that official SWAT could run to end means underflow is ignored, i.e. the compilation flag for underflow is turned off. As mentioned in previous post SWAT Makefile Updated – Stop running when overflow happens, following flag is used to stop SWAT running when invalid, zero, overflow or underflow happens.

-ffpe-trap=invalid,zero,overflow,underflow

To run Eclipse-SWAT successfully, only need to remove underflow flag and re-compile the source codes, which will be become as following format.

-ffpe-trap=invalid,zero,overflow

After that, Eclipse-SWAT could run to end and get exactly the same results as official SWAT.

MinGW Installation Guide for SWAT Debugging

Please note that the administrative privilege is not needed in the whole processes. MinGW would not add entries into register system or change the environment variables.

1. Download the automated GUI installer: mingw-get-setup.exe and run. Accept all the default settings and

1. Choose MinGW packages to install. Open MinGW Installation Manager installed in previous step, select Basic Setup in the left panel and choose package mingw32-base and mingw32-gcc-fortran. Then select menu Installation -> Apply Changes to start installation. The installer would download selected package and extract them into the given folder.

1. Go to the installation folder (C:\MinGW\bin) to check mingw32-make.exe and gfortran.exe.

1. Want to compile SWAT outside of Eclipse? Add folder C:\MinGW\bin to the PATH variable.

Debug SWAT in Eclipse – Utilize Makefile

Don’t Want to Go through all these Processes?

No problem. The Eclipse project created in this post is uploaded to Google Code: https://swat-eclipse.googlecode.com/svn/trunk/SWAT_Makefile. Just check out and open in Eclipse. It’s done!

This version also add Make Target, which is a quick way to build and clean the project. Just double-click to build or clean specific version.

It’s recommended to use SVN Plugin for Eclipse to do this work. I would have another post on this topic in the near future. Please stay tuned.

Contents

• Find Fortran Project… Menu
  
• Create an Empty Makefile Project
  
• Setup the Project
  
  • Binary Parser
    
  • Error Parsers
    
  • Fortran Build Configuration
    
• Copy SWAT Source Codes and Generate Makefile
  
• Build Project
  
• Debug/Run SWAT.exe
  

Please note

1. To compile SWAT, MinGW needs to be installed first. Please refer to another post MinGW Installation Guide for SWAT Debugging
2. Eclipse KEPLER and PTP 7.0.3 was used in this post. These steps should also be able to be used in other version of Eclipse and PTP. For more information, please refer to my previous post Compile and Run SWAT with Photran + MinGW – the easiest way to remove mathematical errors from SWAT.

A Makefile is created in previous post. To use it in Eclipse (Photran/PTP), instead of Executable (Gnu Fortran on Windows) project, we should use the Makefile project, which would allow use of user-defined Makefile.

This post will show you how to create and configure a SWAT Makefile project to utilize the makefile.

Find Fortran Project… Menu

The Fortran Project is not list in menu File -> New when PTP is used the first time. In this case, select Projects… to open New Project window. In this window, you would find Fortran project under Fotran folder. Select it and click Next to create the project. When it’s done, you will be asked if you want to open Fortran perspective. Select Yes. The Fotran Project option would be list in menu File -> New.

Create an Empty Makefile Project

File -> New -> Fortran Project

The project name is SWAT. Select Empty Project under Makefile project and select — Other Toolchain —. Click finish to create the project..

Setup the Project

Open the Property window through right click on the project and select “Properties” to setup the project.

Binary Parser

In Fotran -> Build -> Setting, select PE Windows Parser for Binary Parsers (I’m working a Windows machine)..

Error Parsers

Select Photran Error Parser for GNU Fortran (gfortran) for Error Parsers and move it to the top. This option is not selected when the project is created.

Fortran Build Configuration

Select Fortran Build in the left panel, un-check Use default build command and change Build Command to mingw32-make.

Add debug/ to build directory as there will be two versions of Makefile, one is located in debug folder and another is located in release folder. This one is for debug version.

Change configuration name from Default to Debug.

The default configuration name is Default. It would be nice to rename it to debug. Click Manage Configurations… to open Manage Configurations window. Select default and click Rename… to change name to Debug.

Create Release Configuration

As mentioned before, there is also a release version of Makefile. It’s necessary to make a configuration for it. Here, we could copy all the settings from debug configuration to save time.

Click New… on Manage Configurations window to open Create New Configuration window. Input name as Release and select to copy settings from Debug configuration.

Change the Build directory to ${workspace_loc:/SWAT}/release/.

Copy SWAT Source Codes and Generate Makefile

Copy all the SWAT source codes to the project folder and refresh the project in Eclipse. It would show all the files. Remember to comment the first line in main.f.

Download the GenerateMakefile program and save it into the project folder. Run it to generate the Makefile for debug and release version.

Build Project

Now, it’s ready to build the project, i.e. compile SWAT source codes. Right-click the project and select Build Project to start compile SWAT source codes. You would see some output information from Console window in the bottom. From there, you would know whether or not the process is finished. All the warnings and errors would also be found in this window.

Debug/Run SWAT.exe

After SWAT is compiled successfully, it’s ready to run SWAT. Before that, there is one thing to do: setup Run/Debug Settings, which could be completed in Properties window.

Select Run/Debug Settings in the left panel and click New… to create a Run/Debug Settings. We will first create the settings for debug version. Same as the build configuration, the setting for release version could be also copied from debug version.

In the Main tab, input Fortran Application as debug\SWAT.exe.

In Arguments table, change Working directory to the model txtinout folder.

In Debugger tab, select MinGW gdb for Debugger and uncheck Stop on startup at: option.

In Common tab, select Debug in Display in favorites menu to show the setting in debug menu.

Click debug menu in the toolbar to start running SWAT.

The output information would be display in the Console window.

If some breakpoints are set, SWAT would stop running on breakpoints and ask if want to change to Debug perspective. Click yes to change to Debug perspective, which would re-arrange the windows to facilitate debugging.

You would has change to check all the variable values in Variables window under debugging mode.

Click Fortran button on the up right corner, Eclipse would go back to Fortran perspective, which is designed for coding.

Click the Debug button in the up right corner to return to Debug perspective.

SWAT Makefile Updated – Stop running when overflow happens

Note: This version is updated in next post SWAT Makefile Updated – Ignore Underflow, where underflow flag is removed.

The Makefile published in previous blog Makefile – Compile SWAT using gfortran without modification is updated. It’s highly recommended to update.

Download Link

Makefile Generator, Debug Makefile, Release Makefile

Updates

1. More gfortran flags are added to make sure the SWAT will stop running when errors happen.

In previous version, SWAT would ignore all the errors (e.g. overflow) and run to the end. It’s impossible to know if there are some errors. For some cases, the results are not safe to use.

The official SWAT would stop whenever an error happens and give where the error comes from. It’s the right way to compile SWAT.

In gfortran, the flag -ffpe-trap could used to do this work. The following command flag is added to the Makefile, which would stop the program whenever invalid, zero, overflow or underflow happens.

-ffpe-trap=invalid,zero,overflow,underflow

2. rm command is used to replace del command in the clean target.
   

The del command could only be called from cmd.exe. Eclipse don’t know where is this command and would fail to clean the project.

rm.exe comes along with MSYS in MinGW (package mingw-developer-toolkit and msys-base) and is usually located at C:\MinGW\msys\1.0\bin. Adding this location to the PATH variable in Windows setting or Eclipse project setting would allow Eclipse to call this function when “Clear Project” is used.