jump to navigation

Server crash setelah upgrade dari 8.5 ke 8.5.1 karena HTTP Task 18 July 2010

Posted by tathitsa in Uncategorized.
Tags:
trackback

Setelah upgrade Domino 8.5 ke 8.5.1 FP 3 server crash lebih dari 20 kali….akhirnya buka PMR ke IBM deh…sambil menunggu problem solving dari IBM, saya diskusi dengan master2 domino (Mas Wawang & Pak Gunawan Wibowo).

Trace dimulai dari file NSD yang bisa ditemukan di folder domino_dir\IBM_TECHNICAL_SUPPORT

Pertama lihat file NSD pusing juga…tapi isinya sangat informatif sehingga bisa ditrace dengan baik. Headernya saja sudah menunjukkan task dan thread mana yang membuat crash.

INFO: New files added/deleted to/from directory '/san/domdata'
INFO: Generating binary list file ./nsd.domuser/nsd_Release_8.5.1FP3_cache.ins.lst
INFO: Generating cache file ./nsd.domuser/nsd_Release_8.5.1FP3_cache.ins
Server Name : BTGD1
Script Version : /opt/ibm/lotus/notes/latest/sunspa/nsd.sh 8.5.13.9243
Notes Version : Release 8.5.1FP3 May 23, 2010 (32-bit)
Notes Base : 8.51
Data Dir : /san/domdata
Notes Exec Dir : /opt/ibm/lotus/notes/latest/sunspa
Search Path : /opt/ibm/lotus/notes/latest/sunspa /san/domdata /opt/ibm/lotus/notes/85010/sunspa /opt/ibm/lotus/notes/85010/sunspa/res/C /opt/ibm/lotus/notesapi
Library Path :
Debugger : /usr/proc/bin/pstack
Debugger Version: Standard
MEMCHECK Version: Memcheck Version: 8.5.13.9243
Script Dir : /opt/ibm/lotus/notes/latest/sunspa
Host Info : SunOS btgd1 5.10 Generic_127111-02 sun4u sparc SUNW,Sun-Fire-V440
User : domuser (domuser)
Date : Tue Jul 13 12:03:58 SGT 2010
Last file mod : Tue Sep 1 22:08:33 2009
Input arguments : -batch -crashpid 21803 -crashtid 23 -mmap_file /tmp/.NOTESMEM_please_do_not_remove.f805a800 -wrapper
.......

Dari header diatas berarti yang membuat crash adalah process id 21803 dan thread id 23, nah tinggal di search aja pada file tersebut untuk mendapatkan informasi yang lebih detil.

Hasil pencarian untuk process id dibawah ini menunjukkan bahwa process yang membuat crash adalah http.

.........
domuser   ?  .   599      1 -sh -c cd /domdata; /opt/ibm/lotus/bin/server -jc <cinput>coutput
domuser   R  ..   603    599 -sh -c cd /domdata; /opt/ibm/lotus/bin/server -jc <cinput>coutput
domuser   R  ...   683    603 /opt/ibm/lotus/notes/85010/sunspa/scontroller -jc
domuser   R  .... 15744    683 /opt/ibm/lotus/notes/85010/sunspa/server -j
domuser   R  ..... 16118  15744 /opt/ibm/lotus/notes/85010/sunspa/dircat
domuser   R  ..... 16100  15744 /opt/ibm/lotus/notes/85010/sunspa/diiop
domuser   R  ..... 16097  15744 /opt/ibm/lotus/notes/85010/sunspa/adminp
domuser   R  ..... 16082  15744 /opt/ibm/lotus/notes/85010/sunspa/amgr -s
domuser   R  ...... 16175  16082 /opt/ibm/lotus/notes/85010/sunspa/amgr -e 3
domuser   R  ...... 16172  16082 /opt/ibm/lotus/notes/85010/sunspa/amgr -e 2
domuser   R  ...... 16170  16082 /opt/ibm/lotus/notes/85010/sunspa/amgr -e 1
domuser   R  ..... 16081  15744 /opt/ibm/lotus/notes/85010/sunspa/update
domuser   R  ..... 16080  15744 /opt/ibm/lotus/notes/85010/sunspa/router
domuser   R  ..... 16076  15744 /opt/ibm/lotus/notes/85010/sunspa/replica
domuser   R  ..... 16072  15744 /opt/ibm/lotus/notes/85010/sunspa/replica
domuser   R  ..... 16057  15744 /opt/ibm/lotus/notes/85010/sunspa/replica
domuser   R  ..... 16055  15744 /opt/ibm/lotus/notes/85010/sunspa/replica
domuser   R  ..... 16049  15744 /opt/ibm/lotus/notes/85010/sunspa/replica
domuser   R  ..... 15787  15744 /opt/ibm/lotus/notes/85010/sunspa/event
domuser   R  ..... 21803 15744 /opt/ibm/lotus/notes/85010/sunspa/http
..........

cara cepat untuk mencari penyebab crash server adalah dengan melakukan pencarian kata2: panic, fatal dan crash.


......
###################################
###### thread 23/52 :: http, pid=21803, lwp=23, tid=23 ######
###################################
[1]  fc3c4ce4 nanosleep (d6df3850, d6df3848)
[2]  fc6458f8 OSRunExternalScript (d6df3f00, d6df4004, 6a5, fe5f149c, fe90a874, 252) + 538
[3]  fc642bf8 OSFaultCleanupExt (0, fac00000, 0, fe65ff14, 1, d6df44b8) + 5b8
[4]  fc6425f8 OSFaultCleanup (0, 1000, 0, 6eb8c, 1fab634, 6e800) + 18
[5]  fc6037e8 fatal_error (b, ffcfadb4, d6e025f4, fe5f149c, 305000, 552b) + 588
[6]  f06760f4 __1cUcall_chained_handler6FpnJsigaction_ipnHsiginfo_pv_b_ (b, ba7f58, d6df5550, d6df5298, ba7f50, 20008) + 7c
[7]  f01a5238 JVM_handle_solaris_signal (b, d6df5550, d6df5298, 1, 12b8800, fc610474) + 958
[8]  fc3c4a34 __sighndlr (b, d6df5550, d6df5298, f01a48a8, 0, 1) + c
[9]  fc3b9978 call_user_handler (b, ffbffeff, c, 0, fb335000, d6df5298) + 3b8
[10]  fc610474 OSLockReadFRWSemInt (6400d1c0, 552b, 0, 319000, fe5f149c, 44) + 54
[11]  fba2ad90 __1cNShFormsDbDictOGetTimeLastMod6MrnLtagTIMEDATE_2_v_ (6400cd18, d6df7878, d6df7870, fbcc2da4, 617dc, 61400) + 10
[12]  fb81a66c __1cFHaikuPGetLastModified6MpnFNNote_ippvpLpH5_H_ (0, 124bd18, d6df7a90, d6df87c8, d6df7880, d6dfa3c8) + 6ec
[13]  fb84a89c __1cFHaikuOAtFuncDispatchIDispatch6Mp0pnFNNote_ippvpLpH7_H_ (15c, d6df8d28, d6dfe22c, 4, d6dfa228, d6dfa098) + 27c
[14]  fb7f9ab4 __1cFHaikuNGetHaikuDatum6MpnFNNote_ippvpLpH5_H_ (d6df8d28, d6dfe22c, 4, d6dfa228, d6dfa098, d6dfa3c8) + 54
[15]  fb6c0a98 __1cINFormulaNExtensionProc6MHHpLppvpH1_H_ (d6dfbabc, d6dfa228, b9, d6dfa098, d6dfa228, d6dfa3c8) + e38
[16]  fb6bf744 INotesCompExtProc (1f86dc0, d6dfbabc, b9, 4, d6dfa098, d6dfa228) + 44
[17]  fcc39170 __1cSCompGeneralContextNExtensionProc6MrnHCompute_LppnJCompValue_L_4_ (1f85dc0, 1f86dc0, b9, d6dfa440, 4, b9) + 130
[18]  fcc574a0 __1cNExtensionProcHExecute6M_pknJCompValue__ (1f87024, d6dfa440, c, fe6d32f8, 1f87004, 1973f38) + 180
[19]  fcc7d80c __1cOAtFunctionNodePComputeVariants6M_v_ (1f87024, 0, 0, 1, fcc57320, fe6d434c) + 4c
[20]  fcc7e9f8 __1cSMainExpressionNodePComputeVariants6M_v_ (1f86f08, fffffff8, 421509b9, 25699aa6, fe5f149c, fcc7d7c0) + 18
[21]  fcc7ecf0 __1cIRootNodePComputeVariants6M_v_ (1f8705c, fe6dda5c, fe5f149c, fe5f149c, 1f86f08, 1f86f08) + 50
[22]  fcc2cbd8 __1cHComputeEEval6M_v_ (1f86dc0, 3084, 30a4, 1f89e64, 3000, 0) + b8
[23]  fcc2abec NSFComputeEvaluateExt (1f86dc0, 28, d6dfcd24, d6dfba54, 1, 0) + ac
[24]  fb6bd5e0 __1cINFormulaIEvaluate6MrnGNValue_pi_i_ (d6dfbabc, d6dfccb0, 0, 1229d18, fbcc2da4, 28) + a0
[25]  fb8a3be0 __1cJHuDocNoteIAddField6MpnHHuField_pcipkc_v_ (d6dfe22c, 1f859c0, faad4530, 1, 0, 1f85bc0) + 280
[26]  fb9f4448 __1cKShNotesVarFWrite6MpnRIFormOutputStream__v_ (d6dfe304, 1254f44, 1260d18, 1f859c0, faad4530, fff769c3) + 228
[27]  fb9f0e48 __1cMShDictionaryFWrite6MpnRIFormOutputStream__v_ (125b518, d6dfe304, 125ac18, 125b6bc, 20, 8) + 68
[28]  fb91a598 __1cIHuLayoutJLoadItems6MpnRIFormOutputStream__v_ (125bd18, d6dfe304, 125ab18, 125bf24, 8, 5) + 58
[29]  fb8acb30 __1cJHuDocNoteLLoadSubForm6MpCi_v_ (d6dfe22c, 1256940, 0, 73, fbd34f7c, fb91a540) + 50
[30]  fb91a598 __1cIHuLayoutJLoadItems6MpnRIFormOutputStream__v_ (1256d18, d6dfe304, 124fc18, 1256f24, 8, 7) + 58
[31]  fb8acb30 __1cJHuDocNoteLLoadSubForm6MpCi_v_ (d6dfe22c, 1255d40, 0, 68, fbd34f7c, fb91a540) + 50
[32]  fb91a598 __1cIHuLayoutJLoadItems6MpnRIFormOutputStream__v_ (1254d18, d6dfe304, 124fa18, 1254f24, 8, 5) + 58
[33]  fb8a0e54 __1cJHuDocNoteQLoadItemsAndHtml6M_v_ (d6dfe22c, 1254d18, d6dfe22c, fbcc2da4, 11d8, 1) + 14
[34]  fb869bdc __1cFHaikuPHandleDominoCmd6MrnPEditDocumentCmd__i_ (d6dfe224, 1229d18, 0, fbcc2da4, d6dfe22c, 1000) + fc
[35]  fb7f8060 __1cFHaikuJHandleCmd6FpnDCmd_rnOCmdHandlerBase__i_ (1229d18, 10a5b0, 4b74cef4, 124cd18, fbcc2da4, d6dfe224) + 420
[36]  fb597e58 __1cOCmdHandlerBaseKPrivHandle6MpnDCmd_2_i_ (10a5b0, 1229d18, 0, 1400, 1229d18, fbcc2da4) + 98
[37]  fb592e00 __1cKCmdHandlerKPrivHandle6MpnDCmd__i_ (10a5b0, 1229d18, 24e08, 2892180, 11c8, 1000) + 80
[38]  fb592fd4 __1cKCmdHandlerHHandler6FpnDCmd_pv_nNRequestStatus__ (1229d18, 10a5b0, fb592e80, fbccfc08, ce60, 11ac) + 154
[39]  fb620ec8 __1cbCInotesHTTPProcessRequestImpl6FpnS_InotesHTTPrequest__i_ (2120480, ffffffff, 0, 1231d18, fbcc2da4, 2892180) + 8e8
[40]  fb620564 InotesHTTPProcessRequest (2120480, d6dfd768, 21208a4, 20, 0, 0) + 4
[41]  ff273574 __1cPHTInotesRequestOProcessRequest6M_i_ (2120474, a774, a400, ff3250cc, 2120038, 20336e0) + f4
[42]  ff268688 __1cVHTRequestExtContainerOProcessRequest6MnTHTApplicationSpaces__i_ (3, 8c00, 8eb0, 212019c, a, ff267fc0) + 688
[43]  ff28c318 __1cJHTRequestOProcessRequest6M_i_ (1d400, 0, 0, 0, 0, 2120038) + ed8
[44]  ff298b74 __1cJHTSessionMStartRequest6M_v_ (20336e0, 3, ff32df7c, ff3250cc, 2, ff344594) + 494
[45]  ff2b0e04 __1cOHTWorkerThreadMCheckForWork6M_v_ (b3904c, 0, 11, 0, ff33870c, ff3250cc) + 224
[46]  ff2b0af8 __1cOHTWorkerThreadKThreadMain6M_v_ (b3904c, 26c00, 0, 0, b39054, 0) + 78
[47]  ff2a92b4 HTThreadBeginProc (b3904c, ff2b0a80, ff2b1560, ff34bfc4, ff2a9640, 1) + 34
[48]  fc63c31c ThreadWrapper (0, d6e02000, 0, 2878, 2800, c06d4) + fc
[49]  fc3c4908 _lwp_start (0, 0, 0, 0, 0, 0)
.......

Secara tidak sengaja ada user yang memberikan pencerahan, ceritanya user tersebut ingin mengirimkan attachment namun selalu gagal….ternyata pada saat dia klik tombol Send server langsung crash…kecurigaan mengarah kepada kompatibilitas antara browser yang dia pakai dengan iNotes 8.5.1 karena sebelum diupgrade server tidak crash ataupun down selama lebih dari 6 bulan.

Setelah berdiskusi dengan master2 domino akhirnya percobaan demi percobaan saya lakukan untuk mengatasi masalah ini. Hasilnya:
– iNotes/Forms85.nsf from 8.5.1FP3 and mail template 8.5.1 = crashed
– iNotes/Forms85.nsf from 8.5.1 FP3 and mail template 8.5 = crashed
– iNotes/Forms85.nsf from 8.5 and mail template 8.5.1 = crashed
– iNotes/Forms85.nsf from 8.5 and mail template 8.5 = ok
– iNotes/Forms8 from 8.01 and mail template 8.01 = ok

Dan dari informasi engineer PMR didapatkan bahwa memang yang kompatibel dengan iNotes 8.5 adalah IE 6 & IE 7 :

Title:        Notes 8.5 – Web browsers supported
Doc #:        7013133
URL:        http://www.ibm.com/support/docview.wss?uid=swg27013133

Akhirnya pekerjaan untuk mendowngrade semua mail template user dimulai:

1. shutdown server
2. copy iNotes/Forms85.nsf dari Domino 8.5
3. copy template mail85.ntf dari Domino 8.5
4. startup server
5. l convert mail * mail85.ntf
6. l fixup -f
7. l compact -c
8. l updall -r

User dapat terlayani dengan baik dan Administrator dapat tidur dengan pikiran tenang ….😀

Comments»

1. kamali - 22 August 2010

hi..

selamat berkenalan.

ada experience tak tentang Lotus Mobile configuration?
saya ada plan untuk integrate iPhone dgn Lotus Traveler.

kalau ada pengalaman dan tips-tips, minta tunjuk ajar..

terima kasih

tathitsa - 28 March 2011

hi juga, aku pake lotus traveler utk windows mobile dan nokia.
disamping itu aku juga sdh pake Blackberry Enterprise Server utk Dominonya.

ini lagi nyoba2 lotus traveler yg utk android & iOS, ntar kalo sdh siap aku tulis deh

reference dari http://www-01.ibm.com/software/lotus/products/notes/traveler.html


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: