Jump to content
Francis Lane

GPU model failure late in simulation

Recommended Posts


We are running some TUFLOW models using the GPU hardware and HPC solution scheme (Build 2017-09-AC). We found that some of the runs fail without warning (well into the simulation). It seems that they are 'exiting without prompt', so we are unable to view the dos window to see what has caused the models to fail.

We have tried disabling the 'quick edit mode' in the dos window in case the issue was related to the 'TUFLOW pause mid simulation? cause and solution' topic posted by Chris Huxley, but this didn't make any difference.

Three runs (15, 25, and 540 minute) completed without any apparent issues. Two runs (90 and 120 minute) failed, but when re-started they completed successfully. However three longer runs (720, 2400 and 2880 minute) failed and will not complete on re-start.  We have reviewed the .tlf files (both standard .tlf and hpc.tlf).  We appreciate that the adaptive timestep will mask potential instabilities, but there is nothing  in the tlf files to indicate that TUFLOW is having instability issues (i.e. the timesteps are consistent at the time of failure).

We would appreciate some guidance on how to resolve this issue. We are managing the runs via TRIM.

Kind regards,

Francis Lane


Share this post

Link to post
Share on other sites

No need to reply to this one.

We found the issue was related to an un-announced server change by our IT department. Scheduled policy updates were imposed on our modelling machines. This cut network connectivity while TUFLOW was writing results to a network drive.

Shorter runs all missed the regular scheduled policy update, so all completed without issue.

Hope this helps someone else with a similar issue.





Share this post

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Create New...