Few Tips to install Oracle WebCenter 12c (12.2.1.0.0)

Oracle released its long-awaited 12c version of WebCenter on 23/10/2015.

Interesting thing about WebCenter versioning is that Oracle released its first 12c version with released 2 (12cR2=12.2.x) to match their latest FMW versioning scheme. There is no 12c R1 (12.1.x) version for WebCenter.

In this post, we will see how can we install WebCenter Portal and WebCenter Content quickly for production environment.

Prerequisites

               1.       You must have atleast JDK version 1.8.0_51 or above.

               2.       WLS 12c and WCP12c installer must be executed with JDK not with JRE.

               3.       Database 11g XE is not certified but you can use it for development purpose.

               4.       Unlike SOA suite installation, you need to install Weblogic Server 12c (12.2.1) before you install Oracle WebCenter Portal 12c.

              5.       You must install Weblogic Server with Fusion Middleware Infrastructure installer. This installer will also install JRF and EM Fusion Middleware Control which are required by WCP 12c. 

             6.       Please note that “Oracle Weblogic Server Quick Installer for Developer” and “Generic” installer will not work with WebCenter Portal installation.

      Installation

                  1.       Start the FMW infrastructure installer with following command.

java -jar fmw_12.2.1.0.0_infrastructure.jar

2.    No need to create any domain at this stage. Start the WCP & WCC installer with following command:-

java -jar fmw_12.2.1.0.0_wcportal_generic.jar
java -jar fmw_12.2.1.0.0_wccontent_generic.jar

     
2. RCU is also installed when we install FMW Infra. You will rcu.bat file at C:\middleware12c\Infra\oracle_common\bin\.  Process is same to install schema as we had in 11g.

3.       Use the Configuration Wizard to create a domain using the required domain configuration templates.

4.       By selecting RCU Data in Database Configuration Type screen of configuration wizard, it connect to the database and Service Table (STB) schema to automatically retrieve schema information for schemas needed to configure the domain.

Some Issues:

From 12c, NodeManager is available per host or per domain. If you try to start nodemanager per host, you will get following two errors:-

1.       It will throw error “Identity key store file not found” and shutdown the nodemanager.

Solution:

Quick solution is copy the DemoIdentity.jks file from <DomainHome>\security\ to

<MiddlewareHome>\oracle_common\common\nodemanager\security\ for development domain. 

2.       Processing for domain 'wcp_domain' failed due to configuration error: [The domain 'wcp_domain' at 'null' was not registered in the nodemanager.domains file and dynamic domain registration is not supported. Please register the domain in the nodemanager.domains file.

Solution:

Cop the nodemanager.domains file from <DomainHome>\nodemanager\

To <MiddlewareHome>\oracle_common\common\nodemanager

Alternatively, you can register WebLogic domains with Node Manager using the WLST command, nmEnroll.

How to configure Strong Encryption for Website deployed on Weblogic Server?

Assumption: HTTPS is already configured and its using default SSL Version 3 and default Ciphers.

Newer versions of web browsers (e.g. Chrome) are now configured with policies which only allow websites or portal which enforce the strongest encryption technology to be viewed. SSL version 3 is no more secure due to POODLE attack. Most of the browser disabled support for SSL Version 3 by default. 

Google chrome (version 40+) will throw following error if Weblogic server is configured (by default) to use SSL Version 3.

SSL server probably obsolete.

ERR_SSL_FALLBACK_BEYOND_MINIMUM_VERSION

Firefox Version 34+ throw following error in case website is using SSL Version 3:-

Secure Connection Failed

An error occurred during a connection to <DNS>. Cannot communicate securely with peer: no common encryption algorithm(s). (Error code: ssl_error_no_cypher_overlap)

SSL Lab report will show following results if you are using SSLV3:-

Solution:

We need to disable SSL Version 3 and enable TLS protocol. We also need to enable strong ciphers. We can set either of the following system properties in the command-line argument that starts WebLogic Server:

·         weblogic.security.SSL.protocolVersion=TLS1

·         weblogic.security.SSL.minimumProtocolVersion=TLS1

I would suggest to use 2^nd option. This property value enables any protocol starting with "TLS" for messages that are sent and accepted; for example, TLS V1.0, TLS V1.1, and TLS V1.2.

Implementation in Weblogic Server:-

1.     Enable “Use JSSE SSL” option from WLS Console. For this purpsoe go to Servers> AdminServer or Managed Server >>Configuration>>SSL [Advance Tab]

2.     Now Shutdown domain

3.     Create a custom environment setting file “setCustomEnv.sh” with following lines:-

JAVA_OPTIONS="${JAVA_OPTIONS} -Dweblogic.security.SSL.minimumProtocolVersion=TLSv1.0"

export JAVA_OPTIONS

4.     Include setCustomEnv.sh file into $DOMAIN_HOME/bin/setDomainEnv.sh file:-

if [ "${WEBLOGIC_EXTENSION_DIRS}" != "" ] ; then

                JAVA_OPTIONS="${JAVA_OPTIONS} -Dweblogic.ext.dirs=${WEBLOGIC_EXTENSION_DIRS}"

                export JAVA_OPTIONS

fi

. ${DOMAIN_HOME}/bin/setCustomEnv.sh

JAVA_OPTIONS="${JAVA_OPTIONS}"

export JAVA_OPTIONS

5.     Add following lines in $DOMAIN_HOME/config/config.xml
<server>

<name>AdminServer</name>
<ssl>

      <enabled>true</enabled>

      <ciphersuite>TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA</ciphersuite>

      <ciphersuite>TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA</ciphersuite>

      <ciphersuite>TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA</ciphersuite>

      <ciphersuite>TLS_RSA_WITH_AES_128_CBC_SHA</ciphersuite>

      <ciphersuite>TLS_RSA_WITH_AES_256_CBC_SHA</ciphersuite>

      <ciphersuite>TLS_RSA_WITH_3DES_EDE_CBC_SHA</ciphersuite>

      <ciphersuite>TLS_RSA_WITH_AES_128_CBC_SHA256</ciphersuite>

      <ciphersuite>TLS_RSA_WITH_AES_256_CBC_SHA256</ciphersuite>          

      <hostname-verifier xsi:nil="true"></hostname-verifier>

6.     In order to support strong ciphers (AES 256), we need to download install JCE package for JDK. Download the JCE package from the following location:
http://www.oracle.com/technetwork/java/javase/downloads/jce-7-download-432124.html

7.     Extract the downloaded zip and place the two .jar files US_export_policy.jar and local_policy.jar into your $JDK_HOME/jre/lib/security folder and overwrite old files.

8.     Start the domain

9.     Once server started, verify the site on different browser. I tested on Chrome version 45

10.  Clear the cache of the SSL lab site and scan the website again. You will see A-. Not bad J

You can configure strong ciphers using above method in any oracle product as far its using Weblogic Server.

Note: Starting with the January 20, 2015 Critical Patch Update releases (JDK 8u31, JDK 7u75, JDK 6u91 and above) the Java Runtime Environment has SSLv3 disabled by default.

Reference: http://www.oracle.com/technetwork/java/javase/documentation/cve-2014-3566-2342133.html

How to configure SSL for Admin server when configured multiple authenticators

Symptoms:

We are getting following errors when we enabled SSL port and disabled non-SSL port for Admin server and try to check-in document in WCC:-

<oracle.ods.virtualization.engine.backend.jndi.DefaultAuthenticator.BackendJNDI> <LIBOVD-60143> <[#DefaultAuthenticator]  Unable to create connection to ldap://[localhost]:7002 as null.

javax.naming.CommunicationException: simple bind failed: localhost:7002 [Root exception is javax.net.ssl.SSLException: java.lang.RuntimeException:

Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty]

        at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:218)

         at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2740)

Cause: 

When we configure Admin Server to communicate over SSL then defaultLDAP automatically configure to communicate over SSL protocol (LDAPS). In case of multiple authenticators, and one of them is configure to communicate over SSL, we need to put the corresponding LDAP server's root certificate in an additional keystore used by the virtualisation (LibOVD) functionality otherwise application is unable to establish connection with defaultLDAP (ldaps://localhost:7002).

Solution:
Pre-requisites: Before completing this task, make sure the following configurations:-

• Configure the custom property called virtualize, and set its value to true.
• Admin Server SSL port is not enabled. (*) 
• Managed servers are down 

Steps:

1. Create the keystore:

a.      Set environment variables ORACLE_HOME, WL_HOME and JAVA_HOME.

export ORACLE_HOME=/u01/app/oracle/product/middleware/WC1

export WL_HOME=/u01/app/oracle/product/middleware/wlserver_10.3

export JAVA_HOME=/u01/app/jdk1.7.0_65/

b.      Setup the keystore by running libovdconfig.sh using -createKeystore option.

Open a shell prompt and change the directory to <MW_HOME>/oracle_common/bin. Then, run the following command:

./libovdconfig.sh -host <AdminServerHostName -port 7001 -userName weblogic -domainPath  <DomainPath> –createKeystore

      Enter AdminServer password:[Enter weblogic password]
Enter OVD Keystore password:[Enter a new password to secure a Keystore file]

Once this command runs, we see two new credentials in the Credential Store and a new Keystore file called adapters.jks under <DOMAIN_HOME>\config\fmwconfig\ovd\default\keystores.

2. Export the root certificate from the Admin Server SSL certificate or LDAP directory.
3. Import the root certificate to the libOVD keystore using the keytool command:

$JAVA_Home/bin/keytool -import -keystore adapters.jks -storepass <KeyStore password> -alias <alias of your choice> -file <Admin Server Certificate filename>

4. Enable SSL port and disabled non-SSL port of Admin server. (*)
5. Restart Admin Server and start all the managed servers.

* No need for this step if we are just configuring multiple authenticators with non-defaultLDAP (AD/OID) is communicating over SSL

Reference : Above steps are documented in Oracle BI security Guide: http://docs.oracle.com/cd/E21764_01/bi.1111/e10543/ssl.htm#BIESC6078

How to propagate ECID from OHS to Weblogic Server

What is ECID

Execution Context ID (ECID) is a unique value assigned to each request processed by the Application Server and is used for any subsequent processing spawned from that initial request. With so many discrete components in Fusion Applications this single value is what can be used to tie the processing flow together.

After executing steps mentioned below, you will be able to see single ECID for one request in all the layers (OHS, Weblogic, WebCenter Portal, SOA etc) 

Enable the ECID and cookies logging in OHS

a. Make a backup of the httpd.conf file (in case you haven't).

This file should be located here:
[WEBTIER_INSTANCE]/config/OHS/ohs1

For example, in my case:
/u01/app/oracle/product/Middleware/11.1.1.6/asinst_2/config/OHS/ohs1

b. Edit the httpd.conf file.

c. Locate the "LogFormat" directive.

It should look like this:

LogFormat "%h %l %u %t \"%r\" %>s %b" common

d. Modify the section as follows to include the Oracle ECID and timing metric:

LogFormat "%h %l %u %t \"%r\" %>s %b ecid:%E:%R Cookie=%{Cookie}i Set-Cookie=%{Set-Cookie}o [%T (secs)]" common

e. Save the changes.

f. Stop the OHS

g. Clean all the logs from the: /data/app/admin/ohs1/diagnostics/logs/OHS/ohs1/*

h. Start the OHS.

Verification:

tail –f access_log

172.21.16.1 - - [17/Feb/2015:16:01:13 +1100] [ecid: 0053o6N^T6NBX7P6yfyWMG0007iY00000E:0] [ecid: -] Cookie=JSESSIONID=X9KVxQQM3cfvjuYVERKYIYn3OO49-ydBezDUpVHktP7dX7uMt3IW!-647353198; _WL_AUTHCOOKIE_JSESSIONID=CKUfa0-SVk3CPMHPIpgl; _ga=GA1.3.101510753.1424135093; __utmc=5569967; __utmz=5569967.1424142043.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); BIGipServerADP_WEBCTR_PORTAL_TEST2_tcp8892_pool=1376851372.48162.0000; __utma=5569967.101510753.1424135093.1424142043.1424145235.2; __utmb=5569967.17.10.1424145235; BIGipServerADP_TEST2_http_pool=1628509612.20480.0000 Set-Cookie=- "GET /portal/faces/oracle/webcenter/portalapp/pages/view1.jspx?_afrLoop=2157114934181138&_afrWindowMode=0&_afrWindowId=null HTTP/1.1" 200 6889 [0 (secs)]

Enable the ECID and Cookie logging in WLS (WCP, SOA, WCC, BPM etc)

To see the ECID on the WLS access.log, it helps to correlate better the logs and the flow.

a. Navigate to the WLS Console

b. Select Environment->Servers

c. Select the server you wish to configure

d. Click the "Logging" tab

e. Click the "HTTP" tab

f. Expand the "Advanced" section

g. Set in the "Format field" from the dropdown box "Extended", not "Common" (you may need to click on the "Lock & Edit" button - for Production mode)

h. Add "s-ip c-ip sc(X-ORACLE-DMS-ECID)" and/or "sc(Set-Cookie) cs(Cookie) time-taken" to the Extended Logging Format Fields  (by default is: date time cs-method cs-uri sc-status)

Example:

From : date time cs-method cs-uri sc-status

To:
s-ip c-ip date time sc(X-ORACLE-DMS-ECID) cs-method cs-uri sc-status sc(Set-Cookie) cs(Cookie) time-taken

i. Click the "Save" button (you may need to click on the "Activate Changes" button - for Product mode)

j. Clean all the logs from the: <DOMAIN-HOME>/servers/<MANAGED-SERVER>/logs/*

k. Restart the PORTAL managed server.

Validation :

[adpt2@adptlwc201 logs]$ tail -f access.log

#Version:       1.0

#Fields:        s-ip c-ip date time sc(X-ORACLE-DMS-ECID) cs-method cs-uri sc-status sc(Set-Cookie) cs(Cookie) time-taken

#Software:      WebLogic

#Start-Date:    2015-02-11      15:21:07

RRWC101:8892     172.26.22.24  2015-02-11      15:21:07        -       GET     /       404     -       "__utma=79138256.2059891832.1403589756.1404189723.1405037343.3; lbsid=S05Iu6ltfn87UhA25dsM265yeH-GMj_5yL47KRVzNmtKybXhjNi2!964376355!1422854433134; _ga=GA1.3.2059891832.1403589756" 0.033

Note: Please note that cookies-based-persistence is configured for top URL in the F5.

You can configure above tasks for Weblogic Server, WebCenter Portal, WebCenter Content, SOA Suite and so on as far as it has Weblogic server as a middleware. 

Scaling support for different Oracle Fusion Middleware products

Product	Scale-up	Scale-out	Details
Oracle WebCenter Portal	Yes	Yes	Scale-up is only supported in case of Spaces or custom portal or Portlet managed server.
Oracle WebCenter Content: Content Server	No	Yes
Oracle WebCenter Content: Inbound Refinery	No	Yes
Oracle WebCenter Content: Imaging	Yes	Yes
Oracle WebCenter Content: Capture	Yes	Yes
Oracle SOA/BPM Suite	Yes	Yes	Scale-up is not supported in case of BAM server.
Oracle Service Bus	Yes	Yes

Scale-up: Adding managed server in Existing Nodes

Scale-out: Adding managed servers New Nodes

Reference:

Oracle WebCenter Portal : EDG: http://docs.oracle.com/cd/E29542_01/core.1111/e12037/managing.htm#WCEDG504

Oracle WebCenter Content : EDG: http://docs.oracle.com/cd/E29542_01/doc.1111/e15483/managing.htm#BABGHHHF

Oracle WebCenter SOA : EDG: http://docs.oracle.com/cd/E29542_01/core.1111/e12036/managing.htm#BABGHHHF

Oracle Weblogic Server : Enabling Log rotation of .out file

There is no way to rotate out file from Oracle Weblogic Server Console or EM. To enable log rotation of the out log file on a Linux-based system, perform the following steps:

1. Ensure that you have the logrotate binary installed on your operating system:

ls /usr/sbin/logrotate

2. Create a logrotate configuration file (for example, under /home/oracle/ scripts/logrotate.conf) with the following contents, setting [Domain] to your domain name:

/home/oracle/SOA_DOMAIN/servers/soa_Srv01/logs/soa_Srv01.out {

missingok

copytruncate

compress

rotate=5

size=5M

}

3. Create a logrotate status file (for example, under /home/oracle/scripts/ logrotate.status) with the following contents, setting [Domain] to your domain name:

logrotate state -- version 2

"/home/oracle/SOA_DOMAIN/servers/soa_Srv01/logs/soa_Srv01.out" 2012-1-1

4. Add a new crontab entry by typing crontab -e and adding the following line:

0,15,30,45 * * * * /usr/sbin/logrotate -s /home/oracle/scripts /logrotate.status /home/oracle/scripts/logrotate.conf

This will check every 15 minutes to see if it is necessary to rotate the soa_server1.out standard out file based on the rules defined in the logrotate.conf configuration file. If new log files are added, you must ensure that they are added to the status file or they will not be rotated. This can be repeated for other managed servers as necessary.

You can run the cron job manually to test it.
/usr/sbin/logrotate -s /home/oracle/scripts/logrotate.status /home/oracle/scripts/logrotate.conf

Note: 

You can apply above steps for any Oracle middlware products like Oracle Weblogic Server, WbCenter, SOA, BPM, IDM etc. As this is a linux cron job, I believe its not WLS version specific. 

Right strategy to set startup (JVM+NON-JVM) arguments and debug flags in Weblogic Server

Note: This strategy is applicable in Oracle Weblogic Server, Oracle WebCenter Portal, Oracle SOA/BPM Suite, Oracle WebCenter Content and other Oracle products which are using Weblogic Server. 

Step-by-step guide

1. Finalise the right way
   
   There are two options to configure the JVM arguments:
   1. Via the "Server Start" tab at Administration Console , which is equivalent to the <server-start/> tag in config.xml
   2. Via the script file(startXXX.sh and setXXX.sh) under DOMAIN_HOME/bin/ of each server
   The problem of Option 1 is that the JVM parameters you put in <server-start/> via either Console or directly via config.xml will take effect only if you start the server from NodeManager. If you start the server via scripts, e.g., startWebLogic.sh, the parameters in <server-start/> in config.xml will be ignore.
   For Option 2, if you start servers via scripts under DOMAIN_HOME/bin, specifying JVM parameters will of course work. And as long as you set the StartScriptEnabled=true and StartScriptName=startWebLogic.sh, as required by the Enterprise Deployment Guide, starting servers via NodeManager will also call the same startWebLogic.sh scripts, and hence will pick up the JVM arguments you set there. So you are covered both ways.
   I would go with Option-2 as its covered both ways (nodemanager and startup script), its easy to change via Hudson or manually, its less risky approach as compared to config.xml.
2. Finalise the right script file
   Now, we have chose the right way to specify JVM arguments. Next step is right file to specify the arguments. There are following two files to specify the JVM arguments and non-JVM arguments:-
   1. startWeblogic.sh
   2. setDomain.sh
   setDomain.sh is the right file to specify JVM arguments. This file is called by all startup script like startWeblogic.sh and startManagedServer.sh and nodemanager. Although, startWeblogic.sh is also called by startManagedServer.sh and nodemanager but setDomain.sh is the more logical choice due to its purpose and name.
3. Finalise the right variable
   
   After choosing the right file, next step is choosing the right variables. There are a lot of variables that are related to JVM arguments: MEM_ARGS, USER_MEM_ARGS, JAVA_OPTIONS, JAVA_PROPERTIES etc. Oracle recommend to use the JVM memory settings (i.e. max and min heap size, PermGen size, etc) via variables USER_MEM_ARGS, and specifying any other JVM arguments or non-JVM arguments via JAVA_OPTIONS.
   The advantages of using USER_MEM_ARGS:-
   1. It will override the all standard memory arguments passed to JAVA.
   2. It will reflect right parameters in the server startup logs.
    
4. Finalise the right location in the script file
   It is important to specify the USER_MEM_ARGS in the right location of setDomain.sh file. Following is the right location:-
   
   ==================================================================================
   #IF USER_MEM_ARGS the environment variable is set, use it to override ALL MEM_ARGS values:-
   USER_MEM_ARGS="-server -Xms2g -Xmx2g -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseG1GC -XX:ParallelGCThreads=4"
   export USER_MEM_ARGS 
   if [ "$
   {USER_MEM_ARGS}" != "" ] ; then
   MEM_ARGS="${USER_MEM_ARGS}
   "
   export MEM_ARGS
   fi
   ==================================================================================
5. Finalise the right way to specify the JVM arguments per server
   In such case you can script your JVM settings based on the server names in setDomainEnv.sh file like
   #In case of Admin Server
   if [ "${SERVER_NAME}" == "Admin_Server" ] ; then
   USER_MEM_ARGS="-Xms1024m -Xmx1024m"
   export USER_MEM_ARGS
   fi
   
   #if server name contains WLS_SERVER*if [[ "${SERVER_NAME}" == "WLS_SERVER*"]] ; then
   USER_MEM_ARGS="-server -Xms2g -Xmx2g -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseG1GC -XX:ParallelGCThreads=4"
   export USER_MEM_ARGS
   fi
6. Other recommendations about better management
   a. To be safe, it is better to move the all custom arguments (JVM + NON-JVM) to an external script (e.g. setCustomEnv.sh), and source this external script from setDomainEnv.sh file like:-
    
   -------------------------------------------------------------------------------------------------------------------------------------------------
   #IF USER_MEM_ARGS the environment variable is set, use it to override ALL MEM_ARGS values
    
   . ${DOMAIN_HOME}/bin/setCustomEnv.sh
    
   if [ "${USER_MEM_ARGS}" != "" ] ; then
   MEM_ARGS="${USER_MEM_ARGS}"
   export MEM_ARGS
   fi

         -------------------------------------------------------------------------------------------------------------------------------------------------

        b. To easy to manage and easy to change, we can put the setCustomEnv.sh at a shared file system. So you keep only one copy of it where all the JVM settings are centralised. This single copy of setCustomEnv.sh can be referenced by the individual setDomainEnv.sh for each server.

All done.