Portal 11g Issue: Timing out Fetcher: java.net.SocketTimeoutException; ORA-29273

We are trying to access: http:// portalServer-url:8090/portal/page/portal

We got HTTP 500 output on the page.

 We are trying to access: http:// portalServer-url:8090/portal/page give us the following:

"PPE working" 

This means portal/page is reachable.

(1) check logs:

Portal OHS: ohs1.log

2013-07-17T11:49:57.3141-04:00] [OHS] [NOTIFICATION:16] [OHS-9999] [core.c] [host_id: XXX_HOSTNAME] [host_addr: 192.168.0.1] [pid: 5552] [tid: 484] [user: SYSTEM] [VirtualHost: main]  Child 5552: Child process is exiting

[2013-07-17T11:53:04.2691-04:00] [OHS] [ERROR:32] [OHS-9999] [odl_log.c] [host_id: XXX_HOSTNAME] [host_addr: 192.168.0.1] [pid: 8072] [tid: 1608] [user: SYSTEM] [VirtualHost: portalServer-url:8090]  Request Failed for : /portal/page/portal/Design_Time_PG/Welcome?_esiReqType=3, Resp Code : [500], referer: http://portalServer-url:8090/portal/page/portal/Design_Time_PG/Welcome

[2013-07-17T11:54:12.5823-04:00] [OHS] [ERROR:32] [OHS-9999] [odl_log.c] [host_id: XXX_HOSTNAME] [host_addr: 192.168.0.1] [pid: 8072] [tid: 1624] [user: SYSTEM] [VirtualHost: portalServer-url:8090]  Request Failed for : /portal/page/portal/Design_Time_PG/Welcome?_esiReqType=3, Resp Code : [500], referer: http://portalServer-url:8090/portal/page/portal/Design_Time_PG/Welcome

--------------------------------------------------------------------

WLS_Portal-diagnostic.log

[2013-07-16T16:32:37.729-04:00] [WLS_PORTAL] [ERROR] [] [oracle.portal.page] [tid: content-monitor] [userId: <anonymous>] [ecid: 105607711583,1:25846] [APP: portal] Stall Timeout reached. Interrupting Fetcher name=content-fetcher3 label=pageMeta url=http://portalServer-url:8090/portal/pls/portal/!DEVG_PORTAL.wwpob_page.show/Design_Time_PG/Welcome time=61110ms timeout=61000ms process=GotResponse

[2013-07-16T16:32:37.729-04:00] [WLS_PORTAL] [ERROR] [] [oracle.portal.page] [tid: content-fetcher3] [userId: <anonymous>] [ecid: 105607711583,1:25845] [APP: portal] Connection Timeout reached. Timing out Fetcher: java.net.SocketTimeoutException: Read timed out name=content-fetcher3 label=pageMeta url=http://portalServer-url:8090/portal/pls/portal/!DEVG_PORTAL.wwpob_page.show/Design_Time_PG/Welcome time=61110ms timeout=60000ms process=GotResponse

[2013-07-16T16:32:37.729-04:00] [WLS_PORTAL] [ERROR] [] [oracle.portal.page] [tid: content-monitor] [userId: <anonymous>] [ecid: 105607711583,1:25846] [APP: portal] Unexpected Exception in Monitor during Heartbeat call.[[

java.lang.NullPointerException

               at oracle.webdb.pool.http.HTTPPooledConnectionFactory$Wrapper.stop(Unknown Source)

               at oracle.webdb.dispatcher.HTTPDispatcher.stop(Unknown Source)

               at oracle.webdb.page.ContentFetcher.heartBeat(Unknown Source)

               at oracle.webdb.heartbeat.HeartBeatMonitor.run(Unknown Source)

11g portal is installed. webcache is configured. OHS configured. Checked all servers are up and running.
but http://portalhost_url:8090/portal/page/portal page is not coming up. HTTP 500 status shown.

1 ERROR in ohs.log, 1 in WLS_Portal-diagnostics.log

Path to solution:

(1)I checked the logs in our portal schema and found the following:
Eorror while opening invalidation connection: ORA-29273: HTTP request failed.
ORA-06512: at "SYS.UTL_HTTP", line 1130
ORA-12535: TNS: operation timed out

the above implied that from database, it is having trouble reaching the invalidation url portalhost_url:8093.

(2) I then tried the following in the database schema:  portal

select utl_http.request('http://porthost_url:8093  ') from sys.dual;

I got:  

I get ORA-12541: TNS:no listener

Asked to check network flow and found that port 8093 in portal host is not open to database server.