Right strategy to set startup (JVM+NON-JVM) arguments and debug flags in Weblogic Server

Note: This strategy is applicable in Oracle Weblogic Server, Oracle WebCenter Portal, Oracle SOA/BPM Suite, Oracle WebCenter Content and other Oracle products which are using Weblogic Server. 

Step-by-step guide

1. Finalise the right way
   
   There are two options to configure the JVM arguments:
   1. Via the "Server Start" tab at Administration Console , which is equivalent to the <server-start/> tag in config.xml
   2. Via the script file(startXXX.sh and setXXX.sh) under DOMAIN_HOME/bin/ of each server
   The problem of Option 1 is that the JVM parameters you put in <server-start/> via either Console or directly via config.xml will take effect only if you start the server from NodeManager. If you start the server via scripts, e.g., startWebLogic.sh, the parameters in <server-start/> in config.xml will be ignore.
   For Option 2, if you start servers via scripts under DOMAIN_HOME/bin, specifying JVM parameters will of course work. And as long as you set the StartScriptEnabled=true and StartScriptName=startWebLogic.sh, as required by the Enterprise Deployment Guide, starting servers via NodeManager will also call the same startWebLogic.sh scripts, and hence will pick up the JVM arguments you set there. So you are covered both ways.
   I would go with Option-2 as its covered both ways (nodemanager and startup script), its easy to change via Hudson or manually, its less risky approach as compared to config.xml.
2. Finalise the right script file
   Now, we have chose the right way to specify JVM arguments. Next step is right file to specify the arguments. There are following two files to specify the JVM arguments and non-JVM arguments:-
   1. startWeblogic.sh
   2. setDomain.sh
   setDomain.sh is the right file to specify JVM arguments. This file is called by all startup script like startWeblogic.sh and startManagedServer.sh and nodemanager. Although, startWeblogic.sh is also called by startManagedServer.sh and nodemanager but setDomain.sh is the more logical choice due to its purpose and name.
3. Finalise the right variable
   
   After choosing the right file, next step is choosing the right variables. There are a lot of variables that are related to JVM arguments: MEM_ARGS, USER_MEM_ARGS, JAVA_OPTIONS, JAVA_PROPERTIES etc. Oracle recommend to use the JVM memory settings (i.e. max and min heap size, PermGen size, etc) via variables USER_MEM_ARGS, and specifying any other JVM arguments or non-JVM arguments via JAVA_OPTIONS.
   The advantages of using USER_MEM_ARGS:-
   1. It will override the all standard memory arguments passed to JAVA.
   2. It will reflect right parameters in the server startup logs.
    
4. Finalise the right location in the script file
   It is important to specify the USER_MEM_ARGS in the right location of setDomain.sh file. Following is the right location:-
   
   ==================================================================================
   #IF USER_MEM_ARGS the environment variable is set, use it to override ALL MEM_ARGS values:-
   USER_MEM_ARGS="-server -Xms2g -Xmx2g -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseG1GC -XX:ParallelGCThreads=4"
   export USER_MEM_ARGS 
   if [ "$
   {USER_MEM_ARGS}" != "" ] ; then
   MEM_ARGS="${USER_MEM_ARGS}
   "
   export MEM_ARGS
   fi
   ==================================================================================
5. Finalise the right way to specify the JVM arguments per server
   In such case you can script your JVM settings based on the server names in setDomainEnv.sh file like
   #In case of Admin Server
   if [ "${SERVER_NAME}" == "Admin_Server" ] ; then
   USER_MEM_ARGS="-Xms1024m -Xmx1024m"
   export USER_MEM_ARGS
   fi
   
   #if server name contains WLS_SERVER*if [[ "${SERVER_NAME}" == "WLS_SERVER*"]] ; then
   USER_MEM_ARGS="-server -Xms2g -Xmx2g -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseG1GC -XX:ParallelGCThreads=4"
   export USER_MEM_ARGS
   fi
6. Other recommendations about better management
   a. To be safe, it is better to move the all custom arguments (JVM + NON-JVM) to an external script (e.g. setCustomEnv.sh), and source this external script from setDomainEnv.sh file like:-
    
   -------------------------------------------------------------------------------------------------------------------------------------------------
   #IF USER_MEM_ARGS the environment variable is set, use it to override ALL MEM_ARGS values
    
   . ${DOMAIN_HOME}/bin/setCustomEnv.sh
    
   if [ "${USER_MEM_ARGS}" != "" ] ; then
   MEM_ARGS="${USER_MEM_ARGS}"
   export MEM_ARGS
   fi

         -------------------------------------------------------------------------------------------------------------------------------------------------

        b. To easy to manage and easy to change, we can put the setCustomEnv.sh at a shared file system. So you keep only one copy of it where all the JVM settings are centralised. This single copy of setCustomEnv.sh can be referenced by the individual setDomainEnv.sh for each server.

All done.