Discussion:
[Fastboot] /var/log/messages doesn't have crash info when kernel gets panic/oops/crash
Dharmosoth Seetharam
2009-06-12 09:25:49 UTC
Permalink
Hi,

I have encounter a problem called /var/log/messages does not contain any information
on kernel crash dump.

Prior to enabling KEXEC we saw stack traces and bug info etc in /var/log/messages.
After enabling KEXEC feature those info was missed.

How can we get those missed info in /var/log/messages even when KEXEC feature is enabled.

Any help will appreciate.


Thanks and Regards
Seetharam

____________________________________________________________
Receive Notifications of Incoming Messages
Easily monitor multiple email accounts & access them with a click.
Visit http://www.inbox.com/notifier and check it out!
Neil Horman
2009-06-12 20:05:13 UTC
Permalink
Post by Dharmosoth Seetharam
Hi,
I have encounter a problem called /var/log/messages does not contain any information
on kernel crash dump.
Prior to enabling KEXEC we saw stack traces and bug info etc in /var/log/messages.
After enabling KEXEC feature those info was missed.
How can we get those missed info in /var/log/messages even when KEXEC feature is enabled.
Any help will appreciate.
run crash on the resultant vmcore file that kexec can produce/save and get the
dmesg log out of it.
alternatively you can use makedumpfile with the --dump-dmesg log

Neil
Post by Dharmosoth Seetharam
Thanks and Regards
Seetharam
____________________________________________________________
Receive Notifications of Incoming Messages
Easily monitor multiple email accounts & access them with a click.
Visit http://www.inbox.com/notifier and check it out!
_______________________________________________
kexec mailing list
kexec at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
Dharmosoth Seetharam
2009-06-13 07:01:35 UTC
Permalink
Hi Neil,

Thanks for your quick reply.

What you have suggested is nice when we got/save crash dump (i.e vmcore)
What if we failed to save/capture vmcore info or vmcore is corrupted etc. then we dont get any information
regarding the panic/crash/oops at anywhere?




Thanks and Regards

Seetharam
-----Original Message-----
From: nhorman at redhat.com
Sent: Fri, 12 Jun 2009 16:05:13 -0400
To: dseetharam at inbox.com
Subject: Re: /var/log/messages doesn't have crash info when kernel gets
panic/oops/crash
Post by Dharmosoth Seetharam
Hi,
I have encounter a problem called /var/log/messages does not contain any
information
on kernel crash dump.
Prior to enabling KEXEC we saw stack traces and bug info etc in
/var/log/messages.
After enabling KEXEC feature those info was missed.
How can we get those missed info in /var/log/messages even when KEXEC
feature is enabled.
Any help will appreciate.
run crash on the resultant vmcore file that kexec can produce/save and
get the
dmesg log out of it.
alternatively you can use makedumpfile with the --dump-dmesg log
Neil
Post by Dharmosoth Seetharam
Thanks and Regards
Seetharam
____________________________________________________________
Receive Notifications of Incoming Messages
Easily monitor multiple email accounts & access them with a click.
Visit http://www.inbox.com/notifier and check it out!
_______________________________________________
kexec mailing list
kexec at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
____________________________________________________________
GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at http://www.inbox.com/smileys
Works with AIM?, MSN? Messenger, Yahoo!? Messenger, ICQ?, Google Talk? and most webmails
Eric W. Biederman
2009-06-13 07:11:25 UTC
Permalink
Post by Dharmosoth Seetharam
Hi Neil,
Thanks for your quick reply.
What you have suggested is nice when we got/save crash dump (i.e vmcore)
What if we failed to save/capture vmcore info or vmcore is corrupted etc. then we dont get any information
regarding the panic/crash/oops at anywhere?
Basic questions.
1) Do you have panic on oops set?
I think that setting more than anything else will be the
difference in what shows up in /var/log/messages.
2) Have you tried a dry run and confirmed you can get a crash dump?
3) Are you using kvm


Eric
Post by Dharmosoth Seetharam
Thanks and Regards
Seetharam
-----Original Message-----
From: nhorman at redhat.com
Sent: Fri, 12 Jun 2009 16:05:13 -0400
To: dseetharam at inbox.com
Subject: Re: /var/log/messages doesn't have crash info when kernel gets
panic/oops/crash
Post by Dharmosoth Seetharam
Hi,
I have encounter a problem called /var/log/messages does not contain any
information
on kernel crash dump.
Prior to enabling KEXEC we saw stack traces and bug info etc in
/var/log/messages.
After enabling KEXEC feature those info was missed.
How can we get those missed info in /var/log/messages even when KEXEC
feature is enabled.
Any help will appreciate.
run crash on the resultant vmcore file that kexec can produce/save and
get the
dmesg log out of it.
alternatively you can use makedumpfile with the --dump-dmesg log
Neil
Post by Dharmosoth Seetharam
Thanks and Regards
Seetharam
____________________________________________________________
Receive Notifications of Incoming Messages
Easily monitor multiple email accounts & access them with a click.
Visit http://www.inbox.com/notifier and check it out!
_______________________________________________
kexec mailing list
kexec at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
____________________________________________________________
GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at http://www.inbox.com/smileys
Works with AIM?, MSN? Messenger, Yahoo!? Messenger, ICQ?, Google Talk? and most webmails
Neil Horman
2009-06-13 18:07:36 UTC
Permalink
Post by Dharmosoth Seetharam
Hi Neil,
Thanks for your quick reply.
What you have suggested is nice when we got/save crash dump (i.e vmcore)
What if we failed to save/capture vmcore info or vmcore is corrupted etc. then we dont get any information
regarding the panic/crash/oops at anywhere?
If the vmcore is corrupted to the point where makedumpfile can't extract the
dmesg log, then its certain that syslog wouldn't have been able to write
anything to /var/log/messages either. As a last resort you can try run strings
on the corrupted vmcore and see if the dmesg log is still in there somewhere.

If you forgot to capture the vmcore, theres not much more help I can give you,
Im afraid. You'll just have to make sure you capture it next time. If you only
need the backtrace, and don't want the rest of the vmcore, you can make kdump
just run makedumpfile --dump-dmesg /proc/vmcore /var/log/oops.txt

Neil
Post by Dharmosoth Seetharam
Thanks and Regards
Seetharam
-----Original Message-----
From: nhorman at redhat.com
Sent: Fri, 12 Jun 2009 16:05:13 -0400
To: dseetharam at inbox.com
Subject: Re: /var/log/messages doesn't have crash info when kernel gets
panic/oops/crash
Post by Dharmosoth Seetharam
Hi,
I have encounter a problem called /var/log/messages does not contain any
information
on kernel crash dump.
Prior to enabling KEXEC we saw stack traces and bug info etc in
/var/log/messages.
After enabling KEXEC feature those info was missed.
How can we get those missed info in /var/log/messages even when KEXEC
feature is enabled.
Any help will appreciate.
run crash on the resultant vmcore file that kexec can produce/save and
get the
dmesg log out of it.
alternatively you can use makedumpfile with the --dump-dmesg log
Neil
Post by Dharmosoth Seetharam
Thanks and Regards
Seetharam
____________________________________________________________
Receive Notifications of Incoming Messages
Easily monitor multiple email accounts & access them with a click.
Visit http://www.inbox.com/notifier and check it out!
_______________________________________________
kexec mailing list
kexec at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
____________________________________________________________
GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at http://www.inbox.com/smileys
Works with AIM?, MSN? Messenger, Yahoo!? Messenger, ICQ?, Google Talk? and most webmails
Dharmosoth Seetharam
2009-06-13 10:22:08 UTC
Permalink
-----Original Message-----
From: ebiederm at xmission.com
Sent: Sat, 13 Jun 2009 00:11:25 -0700
To: dseetharam at inbox.com
Subject: Re: /var/log/messages doesn't have crash info when kernel gets
panic/oops/crash
Post by Dharmosoth Seetharam
Hi Neil,
Thanks for your quick reply.
What you have suggested is nice when we got/save crash dump (i.e vmcore)
What if we failed to save/capture vmcore info or vmcore is corrupted
etc. then we dont get any information
regarding the panic/crash/oops at anywhere?
Hi Eric,
Basic questions.
1) Do you have panic on oops set?
I think that setting more than anything else will be the
difference in what shows up in /var/log/messages.
Yes, in my case panic_on_oops was set with 1
2) Have you tried a dry run and confirmed you can get a crash dump?
No, I haven't tried this.
3) Are you using kvm
kvm? kernel virtual memory?

thanks,
Seetharam
Eric
Post by Dharmosoth Seetharam
Thanks and Regards
Seetharam
-----Original Message-----
From: nhorman at redhat.com
Sent: Fri, 12 Jun 2009 16:05:13 -0400
To: dseetharam at inbox.com
Subject: Re: /var/log/messages doesn't have crash info when kernel gets
panic/oops/crash
Post by Dharmosoth Seetharam
Hi,
I have encounter a problem called /var/log/messages does not contain
any
information
on kernel crash dump.
Prior to enabling KEXEC we saw stack traces and bug info etc in
/var/log/messages.
After enabling KEXEC feature those info was missed.
How can we get those missed info in /var/log/messages even when KEXEC
feature is enabled.
Any help will appreciate.
run crash on the resultant vmcore file that kexec can produce/save and
get the
dmesg log out of it.
alternatively you can use makedumpfile with the --dump-dmesg log
Neil
Post by Dharmosoth Seetharam
Thanks and Regards
Seetharam
____________________________________________________________
Receive Notifications of Incoming Messages
Easily monitor multiple email accounts & access them with a click.
Visit http://www.inbox.com/notifier and check it out!
_______________________________________________
kexec mailing list
kexec at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
____________________________________________________________
GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at
http://www.inbox.com/smileys
Works with AIM?, MSN? Messenger, Yahoo!? Messenger, ICQ?, Google Talk?
and most webmails
____________________________________________________________
Receive Notifications of Incoming Messages
Easily monitor multiple email accounts & access them with a click.
Visit http://www.inbox.com/notifier and check it out!
Eric W. Biederman
2009-06-13 12:51:25 UTC
Permalink
Post by Dharmosoth Seetharam
-----Original Message-----
From: ebiederm at xmission.com
Sent: Sat, 13 Jun 2009 00:11:25 -0700
To: dseetharam at inbox.com
Subject: Re: /var/log/messages doesn't have crash info when kernel gets
panic/oops/crash
Post by Dharmosoth Seetharam
Hi Neil,
Thanks for your quick reply.
What you have suggested is nice when we got/save crash dump (i.e vmcore)
What if we failed to save/capture vmcore info or vmcore is corrupted
etc. then we dont get any information
regarding the panic/crash/oops at anywhere?
Hi Eric,
Basic questions.
1) Do you have panic on oops set?
I think that setting more than anything else will be the
difference in what shows up in /var/log/messages.
Yes, in my case panic_on_oops was set with 1
That is the reason thing don't show up in /var/log/messages
because you panic before syslog has a chance to write the messages.
Post by Dharmosoth Seetharam
2) Have you tried a dry run and confirmed you can get a crash dump?
No, I haven't tried this.
It sounds like you have not been getting the coredumps when problems
happen. Running a simple test run to Alt-sysrq-c to confirm
that things are basically setup ok is a good idea.
Post by Dharmosoth Seetharam
3) Are you using kvm
kvm? kernel virtual memory?
Kernel virtual machine. In particular.
Hardware acceleration for virtual machines.

Eric
Dharmosoth Seetharam
2009-06-13 13:24:09 UTC
Permalink
-----Original Message-----
From: ebiederm at xmission.com
Sent: Sat, 13 Jun 2009 05:51:25 -0700
To: dseetharam at inbox.com
Subject: Re: /var/log/messages doesn't have crash info when kernel gets
panic/oops/crash
Post by Dharmosoth Seetharam
-----Original Message-----
From: ebiederm at xmission.com
Sent: Sat, 13 Jun 2009 00:11:25 -0700
To: dseetharam at inbox.com
Subject: Re: /var/log/messages doesn't have crash info when kernel gets
panic/oops/crash
Post by Dharmosoth Seetharam
Hi Neil,
Thanks for your quick reply.
What you have suggested is nice when we got/save crash dump (i.e
vmcore)
What if we failed to save/capture vmcore info or vmcore is corrupted
etc. then we dont get any information
regarding the panic/crash/oops at anywhere?
Hi Eric,
Basic questions.
1) Do you have panic on oops set?
I think that setting more than anything else will be the
difference in what shows up in /var/log/messages.
Yes, in my case panic_on_oops was set with 1
That is the reason thing don't show up in /var/log/messages
because you panic before syslog has a chance to write the messages.
Thanks.
So, if panic_on_oops set with 0(default), we will get all the
needed info like stack traces, bug info etc. in /var/log/messages then reboots.

Or do we need to apply any specific patches to get info in /var/log/messages?
Post by Dharmosoth Seetharam
2) Have you tried a dry run and confirmed you can get a crash dump?
No, I haven't tried this.
It sounds like you have not been getting the coredumps when problems
happen. Running a simple test run to Alt-sysrq-c to confirm
that things are basically setup ok is a good idea.
Sorry, I misunderstood your question.
I have gave dry run and confirmed that the dumps are getting saved in particular dir and able to analyze.

I did in both the ways like
1 - echo c > /proc/sysrq-trigger
2 -Alt -sysrq -c



thanks,
Seetharam
Post by Dharmosoth Seetharam
3) Are you using kvm
kvm? kernel virtual memory?
Kernel virtual machine. In particular.
Hardware acceleration for virtual machines.
Eric
____________________________________________________________
FREE ONLINE PHOTOSHARING - Share your photos online with your friends and family!
Visit http://www.inbox.com/photosharing to find out more!
Eric W. Biederman
2009-06-13 14:08:50 UTC
Permalink
Post by Dharmosoth Seetharam
Post by Eric W. Biederman
Post by Dharmosoth Seetharam
Post by Eric W. Biederman
Basic questions.
1) Do you have panic on oops set?
I think that setting more than anything else will be the
difference in what shows up in /var/log/messages.
Yes, in my case panic_on_oops was set with 1
That is the reason thing don't show up in /var/log/messages
because you panic before syslog has a chance to write the messages.
Thanks.
So, if panic_on_oops set with 0(default), we will get all the
needed info like stack traces, bug info etc. in /var/log/messages then reboots.
Or do we need to apply any specific patches to get info in /var/log/messages?
Post by Eric W. Biederman
Post by Dharmosoth Seetharam
Post by Eric W. Biederman
2) Have you tried a dry run and confirmed you can get a crash dump?
No, I haven't tried this.
It sounds like you have not been getting the coredumps when problems
happen. Running a simple test run to Alt-sysrq-c to confirm
that things are basically setup ok is a good idea.
Sorry, I misunderstood your question.
I have gave dry run and confirmed that the dumps are getting saved in particular dir and able to analyze.
I did in both the ways like
1 - echo c > /proc/sysrq-trigger
2 -Alt -sysrq -c
Then unless you are having problems capturing core dumps in real
failure situations it sounds like all is well with the world.

A kernel oops is normally not fatal and the kernel tries to limp along
allowing for better diagnostics etc. This allows klogd to read the
kernels message buffer and pass the kernel messages to syslog which
writes the messages to /var/log/messages.

After a kernel panic nothing is allowed to happen which prevents
user space from writing to /var/log/messages in the usual way.

If you want data in /var/log/messages it appears you have two choices.
1) Post process a core dump as Neil suggested and feed the kernel log
buffer to syslog.
2) disable panic on oops. The system will continue to limp along allowing
user space to write to /var/log/messages.

Eric
Dharmosoth Seetharam
2009-06-13 15:08:58 UTC
Permalink
-----Original Message-----
From: ebiederm at xmission.com
Sent: Sat, 13 Jun 2009 07:08:50 -0700
To: dseetharam at inbox.com
Subject: Re: /var/log/messages doesn't have crash info when kernel gets
panic/oops/crash
Post by Dharmosoth Seetharam
Post by Eric W. Biederman
Post by Dharmosoth Seetharam
Post by Eric W. Biederman
Basic questions.
1) Do you have panic on oops set?
I think that setting more than anything else will be the
difference in what shows up in /var/log/messages.
Yes, in my case panic_on_oops was set with 1
That is the reason thing don't show up in /var/log/messages
because you panic before syslog has a chance to write the messages.
Thanks.
So, if panic_on_oops set with 0(default), we will get all the
needed info like stack traces, bug info etc. in /var/log/messages then
reboots.
Or do we need to apply any specific patches to get info in
/var/log/messages?
Post by Eric W. Biederman
Post by Dharmosoth Seetharam
Post by Eric W. Biederman
2) Have you tried a dry run and confirmed you can get a crash dump?
No, I haven't tried this.
It sounds like you have not been getting the coredumps when problems
happen. Running a simple test run to Alt-sysrq-c to confirm
that things are basically setup ok is a good idea.
Sorry, I misunderstood your question.
I have gave dry run and confirmed that the dumps are getting saved in
particular dir and able to analyze.
I did in both the ways like
1 - echo c > /proc/sysrq-trigger
2 -Alt -sysrq -c
Then unless you are having problems capturing core dumps in real
failure situations it sounds like all is well with the world.
A kernel oops is normally not fatal and the kernel tries to limp along
allowing for better diagnostics etc. This allows klogd to read the
kernels message buffer and pass the kernel messages to syslog which
writes the messages to /var/log/messages.
After a kernel panic nothing is allowed to happen which prevents
user space from writing to /var/log/messages in the usual way.
If you want data in /var/log/messages it appears you have two choices.
1) Post process a core dump as Neil suggested and feed the kernel log
buffer to syslog.
2) disable panic on oops. The system will continue to limp along
allowing
user space to write to /var/log/messages.
Eric
Thanks Eric.

I will try all the ways as you suggested and let you know the outcome.

regards,
Seetharam

____________________________________________________________
FREE ONLINE PHOTOSHARING - Share your photos online with your friends and family!
Visit http://www.inbox.com/photosharing to find out more!
Dharmosoth Seetharam
2009-06-16 04:43:02 UTC
Permalink
Hi,

I have experimented few things and results are below.

Case 1) echo 0 > /proc/sys/kernel/panic_on_oops
When BUG/oops occurred it is able to log the diagnostics to /var/log/messages
and then continue(not switching to dump-saving kernel).

What I am looking is - I want both the things to be happen.
that is - it should log the appropriate messages in /var/log/messsages and
should be able to take kernel dump.

case 2) In general, When panic has occurred we won't log any type of messages in
/var/log/messages ... am I right ?
If KEXEC is enabled and dump-saving kernel is loaded then
will switch to it and save the kernel dump.

I have added the dump_stack() function call in the panic() function
before calling crash_kexec with few seconds of delay. It is behaving
like randomly (ie. some time stack traces are logged in /var/log/messages and
not for rest of the time)

What I am looking is - I want to log the appropriate messages in /var/log/messages
and then switch to dump-saving kernel.

case 3) In general, When die/nmi_die called we used to log the appropriate messages in /var/log/message
and then saving kernel dump will be depends on KEXEC enable and loading dump-saving kernel.
--- am I right ?

I have added the delay in die() function with few seconds before calling crash_kexec()
But I never seen messages logged in /var/log/messages, I can see those on serial console.

Here also my intention is want both, ie. log the appropriate messages in /var/log/messages
and then take the action as per KEXEC enabled.



Please advice me, What we can do to achieve those things.

Thanks for helping me here.


regards,
Seetharam
-----Original Message-----
From: nhorman at redhat.com
Sent: Sat, 13 Jun 2009 14:07:36 -0400
To: dseetharam at inbox.com
Subject: Re: /var/log/messages doesn't have crash info when kernel gets
panic/oops/crash
Post by Dharmosoth Seetharam
Hi Neil,
Thanks for your quick reply.
What you have suggested is nice when we got/save crash dump (i.e vmcore)
What if we failed to save/capture vmcore info or vmcore is corrupted
etc. then we dont get any information
regarding the panic/crash/oops at anywhere?
If the vmcore is corrupted to the point where makedumpfile can't extract
the
dmesg log, then its certain that syslog wouldn't have been able to write
anything to /var/log/messages either. As a last resort you can try run
strings
on the corrupted vmcore and see if the dmesg log is still in there
somewhere.
If you forgot to capture the vmcore, theres not much more help I can give
you,
Im afraid. You'll just have to make sure you capture it next time. If
you only
need the backtrace, and don't want the rest of the vmcore, you can make
kdump
just run makedumpfile --dump-dmesg /proc/vmcore /var/log/oops.txt
Neil
Post by Dharmosoth Seetharam
Thanks and Regards
Seetharam
-----Original Message-----
From: nhorman at redhat.com
Sent: Fri, 12 Jun 2009 16:05:13 -0400
To: dseetharam at inbox.com
Subject: Re: /var/log/messages doesn't have crash info when kernel gets
panic/oops/crash
Post by Dharmosoth Seetharam
Hi,
I have encounter a problem called /var/log/messages does not contain
any
information
on kernel crash dump.
Prior to enabling KEXEC we saw stack traces and bug info etc in
/var/log/messages.
After enabling KEXEC feature those info was missed.
How can we get those missed info in /var/log/messages even when KEXEC
feature is enabled.
Any help will appreciate.
run crash on the resultant vmcore file that kexec can produce/save and
get the
dmesg log out of it.
alternatively you can use makedumpfile with the --dump-dmesg log
Neil
Post by Dharmosoth Seetharam
Thanks and Regards
Seetharam
____________________________________________________________
Receive Notifications of Incoming Messages
Easily monitor multiple email accounts & access them with a click.
Visit http://www.inbox.com/notifier and check it out!
_______________________________________________
kexec mailing list
kexec at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
____________________________________________________________
GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at
http://www.inbox.com/smileys
Works with AIM?, MSN? Messenger, Yahoo!? Messenger, ICQ?, Google Talk?
and most webmails
Haren Myneni
2009-06-16 05:43:45 UTC
Permalink
fastboot-bounces at lists.linux-foundation.org wrote on 06/15/2009 09:43:02
Dharmosoth Seetharam <dseetharam at inbox.com>
Hi,
I have experimented few things and results are below.
Case 1) echo 0 > /proc/sys/kernel/panic_on_oops
When BUG/oops occurred it is able to log the diagnostics to /var/log/messages
and then continue(not switching to dump-saving kernel).
What I am looking is - I want both the things to be happen.
that is - it should log the appropriate messages in /var/
log/messsages and
should be able to take kernel dump.
when panic_on_oops is not set, it does not call crash_kexec() - change in
kexec_should_crash() in kernel/kexec.c
case 2) In general, When panic has occurred we won't log any type of messages in
/var/log/messages ... am I right ?
If KEXEC is enabled and dump-saving kernel is loaded then
will switch to it and save the kernel dump.
I have added the dump_stack() function call in the panic() function
before calling crash_kexec with few seconds of delay. It is behaving
like randomly (ie. some time stack traces are logged in /
var/log/messages and
not for rest of the time)
What I am looking is - I want to log the appropriate
messages in /var/log/messages
and then switch to dump-saving kernel.
printk messages in /var/log/messages is depends on how fast klogd flushes
printk buffer.
case 3) In general, When die/nmi_die called we used to log the
appropriate messages in /var/log/message
and then saving kernel dump will be depends on KEXEC enable
and loading dump-saving kernel.
--- am I right ?
I have added the delay in die() function with few seconds
before calling crash_kexec()
But I never seen messages logged in /var/log/messages, I can
see those on serial console.
Here also my intention is want both, ie. log the appropriate
messages in /var/log/messages
and then take the action as per KEXEC enabled.
printk write to the console and its buffer, As I said, depends on time for
klogd flushes printk buffer.
Please advice me, What we can do to achieve those things.
Thanks for helping me here.
regards,
Seetharam
-----Original Message-----
From: nhorman at redhat.com
Sent: Sat, 13 Jun 2009 14:07:36 -0400
To: dseetharam at inbox.com
Subject: Re: /var/log/messages doesn't have crash info when kernel gets
panic/oops/crash
Post by Dharmosoth Seetharam
Hi Neil,
Thanks for your quick reply.
What you have suggested is nice when we got/save crash dump (i.e vmcore)
What if we failed to save/capture vmcore info or vmcore is corrupted
etc. then we dont get any information
regarding the panic/crash/oops at anywhere?
If the vmcore is corrupted to the point where makedumpfile can't extract
the
dmesg log, then its certain that syslog wouldn't have been able to write
anything to /var/log/messages either. As a last resort you can try run
strings
on the corrupted vmcore and see if the dmesg log is still in there
somewhere.
If you forgot to capture the vmcore, theres not much more help I can give
you,
Im afraid. You'll just have to make sure you capture it next time. If
you only
need the backtrace, and don't want the rest of the vmcore, you can make
kdump
just run makedumpfile --dump-dmesg /proc/vmcore /var/log/oops.txt
Neil
Post by Dharmosoth Seetharam
Thanks and Regards
Seetharam
-----Original Message-----
From: nhorman at redhat.com
Sent: Fri, 12 Jun 2009 16:05:13 -0400
To: dseetharam at inbox.com
Subject: Re: /var/log/messages doesn't have crash info when kernel gets
panic/oops/crash
Post by Dharmosoth Seetharam
Hi,
I have encounter a problem called /var/log/messages does not contain
any
information
on kernel crash dump.
Prior to enabling KEXEC we saw stack traces and bug info etc in
/var/log/messages.
After enabling KEXEC feature those info was missed.
How can we get those missed info in /var/log/messages even when KEXEC
feature is enabled.
Any help will appreciate.
run crash on the resultant vmcore file that kexec can produce/save and
get the
dmesg log out of it.
alternatively you can use makedumpfile with the --dump-dmesg log
Neil
Post by Dharmosoth Seetharam
Thanks and Regards
Seetharam
____________________________________________________________
Receive Notifications of Incoming Messages
Easily monitor multiple email accounts & access them with a click.
Visit http://www.inbox.com/notifier and check it out!
_______________________________________________
kexec mailing list
kexec at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
____________________________________________________________
GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at
http://www.inbox.com/smileys
Works with AIM?, MSN? Messenger, Yahoo!? Messenger, ICQ?, Google Talk?
and most webmails
_______________________________________________
fastboot mailing list
fastboot at lists.linux-foundation.org
https://lists.linux-foundation.org/mailman/listinfo/fastboot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.linux-foundation.org/pipermail/fastboot/attachments/20090615/c1278e22/attachment.htm
Eric W. Biederman
2009-06-16 06:19:37 UTC
Permalink
Post by Dharmosoth Seetharam
Hi,
I have experimented few things and results are below.
Case 1) echo 0 > /proc/sys/kernel/panic_on_oops
When BUG/oops occurred it is able to log the diagnostics to /var/log/messages
and then continue(not switching to dump-saving kernel).
What I am looking is - I want both the things to be happen.
that is - it should log the appropriate messages in /var/log/messsages and
should be able to take kernel dump.
You can't have it. Sorry.
Keeping user space alive is something that simply can not be done reliably.
Post by Dharmosoth Seetharam
Please advice me, What we can do to achieve those things.
The only scenario I see possible is to take the dump. Use
makedumpfile or the crash log command to extract the kernel messages
and put them in the /var/log/messages.

Eric
Eric W. Biederman
2009-06-16 06:19:33 UTC
Permalink
Post by Dharmosoth Seetharam
Hi,
I have experimented few things and results are below.
Case 1) echo 0 > /proc/sys/kernel/panic_on_oops
When BUG/oops occurred it is able to log the diagnostics to /var/log/messages
and then continue(not switching to dump-saving kernel).
What I am looking is - I want both the things to be happen.
that is - it should log the appropriate messages in /var/log/messsages and
should be able to take kernel dump.
You can't have it. Sorry.
Keeping user space alive is something that simply can not be done reliably.
Post by Dharmosoth Seetharam
Please advice me, What we can do to achieve those things.
The only scenario I see possible is to take the dump. Use
makedumpfile or the crash log command to extract the kernel messages
and put them in the /var/log/messages.

Eric
Neil Horman
2009-06-16 11:24:25 UTC
Permalink
Post by Dharmosoth Seetharam
Hi,
I have experimented few things and results are below.
Case 1) echo 0 > /proc/sys/kernel/panic_on_oops
When BUG/oops occurred it is able to log the diagnostics to /var/log/messages
and then continue(not switching to dump-saving kernel).
What I am looking is - I want both the things to be happen.
that is - it should log the appropriate messages in /var/log/messsages and
should be able to take kernel dump.
case 2) In general, When panic has occurred we won't log any type of messages in
/var/log/messages ... am I right ?
If KEXEC is enabled and dump-saving kernel is loaded then
will switch to it and save the kernel dump.
I have added the dump_stack() function call in the panic() function
before calling crash_kexec with few seconds of delay. It is behaving
like randomly (ie. some time stack traces are logged in /var/log/messages and
not for rest of the time)
What I am looking is - I want to log the appropriate messages in /var/log/messages
and then switch to dump-saving kernel.
Then what you want to do is, enable kdump, and set your kernel up to
panic_on_oops. Then, configure kdump to extract the dmesg log from
/proc/vmcore, and write that to a file (either a file you create, or to the end
of /var/log/messages). Most distributions kdump configurations allow for
something simmilar to this. The new config method I've got in rawhide should
make this very easy.

The problem is that when you panic the box (which is what has to happen for
kdump to start), the syslog daemon doesn't get a chance to run first, so theres
no way to write the oops backtrace to var/log/messages. You'll need to tell
kdump to do that for you after the fact.
Post by Dharmosoth Seetharam
case 3) In general, When die/nmi_die called we used to log the appropriate messages in /var/log/message
and then saving kernel dump will be depends on KEXEC enable and loading dump-saving kernel.
--- am I right ?
I have added the delay in die() function with few seconds before calling crash_kexec()
But I never seen messages logged in /var/log/messages, I can see those on serial console.
See above, when we printk those messages, klogd/syslogd needs to run to get them
from the console buffer and write them to /var/log/messages. Theres no
guarantee that will ever happen.
Post by Dharmosoth Seetharam
Here also my intention is want both, ie. log the appropriate messages in /var/log/messages
and then take the action as per KEXEC enabled.
Please advice me, What we can do to achieve those things.
See above, use makedumpfile to extract the dmesg log from /proc/vmcore in kdump,
then use a custom script to append the log to /var/log/messages. In fedora that
will be equivalent to using the core_collector and kdump_post options in
/etc/kdump.conf.

Neil
Post by Dharmosoth Seetharam
Thanks for helping me here.
regards,
Seetharam
-----Original Message-----
From: nhorman at redhat.com
Sent: Sat, 13 Jun 2009 14:07:36 -0400
To: dseetharam at inbox.com
Subject: Re: /var/log/messages doesn't have crash info when kernel gets
panic/oops/crash
Post by Dharmosoth Seetharam
Hi Neil,
Thanks for your quick reply.
What you have suggested is nice when we got/save crash dump (i.e vmcore)
What if we failed to save/capture vmcore info or vmcore is corrupted
etc. then we dont get any information
regarding the panic/crash/oops at anywhere?
If the vmcore is corrupted to the point where makedumpfile can't extract
the
dmesg log, then its certain that syslog wouldn't have been able to write
anything to /var/log/messages either. As a last resort you can try run
strings
on the corrupted vmcore and see if the dmesg log is still in there
somewhere.
If you forgot to capture the vmcore, theres not much more help I can give
you,
Im afraid. You'll just have to make sure you capture it next time. If
you only
need the backtrace, and don't want the rest of the vmcore, you can make
kdump
just run makedumpfile --dump-dmesg /proc/vmcore /var/log/oops.txt
Neil
Post by Dharmosoth Seetharam
Thanks and Regards
Seetharam
-----Original Message-----
From: nhorman at redhat.com
Sent: Fri, 12 Jun 2009 16:05:13 -0400
To: dseetharam at inbox.com
Subject: Re: /var/log/messages doesn't have crash info when kernel gets
panic/oops/crash
Post by Dharmosoth Seetharam
Hi,
I have encounter a problem called /var/log/messages does not contain
any
information
on kernel crash dump.
Prior to enabling KEXEC we saw stack traces and bug info etc in
/var/log/messages.
After enabling KEXEC feature those info was missed.
How can we get those missed info in /var/log/messages even when KEXEC
feature is enabled.
Any help will appreciate.
run crash on the resultant vmcore file that kexec can produce/save and
get the
dmesg log out of it.
alternatively you can use makedumpfile with the --dump-dmesg log
Neil
Post by Dharmosoth Seetharam
Thanks and Regards
Seetharam
____________________________________________________________
Receive Notifications of Incoming Messages
Easily monitor multiple email accounts & access them with a click.
Visit http://www.inbox.com/notifier and check it out!
_______________________________________________
kexec mailing list
kexec at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
____________________________________________________________
GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at
http://www.inbox.com/smileys
Works with AIM?, MSN? Messenger, Yahoo!? Messenger, ICQ?, Google Talk?
and most webmails
Dharmosoth Seetharam
2009-06-16 11:47:01 UTC
Permalink
Thanks a lot.
Seetharam
-----Original Message-----
From: nhorman at redhat.com
Sent: Tue, 16 Jun 2009 07:24:25 -0400
To: dseetharam at inbox.com
Subject: Re: /var/log/messages doesn't have crash info when kernel gets
panic/oops/crash
Post by Dharmosoth Seetharam
Hi,
I have experimented few things and results are below.
Case 1) echo 0 > /proc/sys/kernel/panic_on_oops
When BUG/oops occurred it is able to log the diagnostics to
/var/log/messages
and then continue(not switching to dump-saving kernel).
What I am looking is - I want both the things to be happen.
that is - it should log the appropriate messages in
/var/log/messsages and
should be able to take kernel dump.
case 2) In general, When panic has occurred we won't log any type of
messages in
/var/log/messages ... am I right ?
If KEXEC is enabled and dump-saving kernel is loaded then
will switch to it and save the kernel dump.
I have added the dump_stack() function call in the panic()
function
before calling crash_kexec with few seconds of delay. It is
behaving
like randomly (ie. some time stack traces are logged in
/var/log/messages and
not for rest of the time)
What I am looking is - I want to log the appropriate messages
in /var/log/messages
and then switch to dump-saving kernel.
Then what you want to do is, enable kdump, and set your kernel up to
panic_on_oops. Then, configure kdump to extract the dmesg log from
/proc/vmcore, and write that to a file (either a file you create, or to
the end
of /var/log/messages). Most distributions kdump configurations allow for
something simmilar to this. The new config method I've got in rawhide
should
make this very easy.
The problem is that when you panic the box (which is what has to happen
for
kdump to start), the syslog daemon doesn't get a chance to run first, so
theres
no way to write the oops backtrace to var/log/messages. You'll need to
tell
kdump to do that for you after the fact.
Post by Dharmosoth Seetharam
case 3) In general, When die/nmi_die called we used to log the
appropriate messages in /var/log/message
and then saving kernel dump will be depends on KEXEC enable and
loading dump-saving kernel.
--- am I right ?
I have added the delay in die() function with few seconds before
calling crash_kexec()
But I never seen messages logged in /var/log/messages, I can see
those on serial console.
See above, when we printk those messages, klogd/syslogd needs to run to
get them
from the console buffer and write them to /var/log/messages. Theres no
guarantee that will ever happen.
Post by Dharmosoth Seetharam
Here also my intention is want both, ie. log the appropriate
messages in /var/log/messages
and then take the action as per KEXEC enabled.
Please advice me, What we can do to achieve those things.
See above, use makedumpfile to extract the dmesg log from /proc/vmcore in
kdump,
then use a custom script to append the log to /var/log/messages. In
fedora that
will be equivalent to using the core_collector and kdump_post options in
/etc/kdump.conf.
Neil
Post by Dharmosoth Seetharam
Thanks for helping me here.
regards,
Seetharam
-----Original Message-----
From: nhorman at redhat.com
Sent: Sat, 13 Jun 2009 14:07:36 -0400
To: dseetharam at inbox.com
Subject: Re: /var/log/messages doesn't have crash info when kernel gets
panic/oops/crash
Post by Dharmosoth Seetharam
Hi Neil,
Thanks for your quick reply.
What you have suggested is nice when we got/save crash dump (i.e
vmcore)
What if we failed to save/capture vmcore info or vmcore is corrupted
etc. then we dont get any information
regarding the panic/crash/oops at anywhere?
If the vmcore is corrupted to the point where makedumpfile can't
extract
the
dmesg log, then its certain that syslog wouldn't have been able to
write
anything to /var/log/messages either. As a last resort you can try run
strings
on the corrupted vmcore and see if the dmesg log is still in there
somewhere.
If you forgot to capture the vmcore, theres not much more help I can
give
you,
Im afraid. You'll just have to make sure you capture it next time. If
you only
need the backtrace, and don't want the rest of the vmcore, you can make
kdump
just run makedumpfile --dump-dmesg /proc/vmcore /var/log/oops.txt
Neil
Post by Dharmosoth Seetharam
Thanks and Regards
Seetharam
-----Original Message-----
From: nhorman at redhat.com
Sent: Fri, 12 Jun 2009 16:05:13 -0400
To: dseetharam at inbox.com
Subject: Re: /var/log/messages doesn't have crash info when kernel
gets
panic/oops/crash
Post by Dharmosoth Seetharam
Hi,
I have encounter a problem called /var/log/messages does not contain
any
information
on kernel crash dump.
Prior to enabling KEXEC we saw stack traces and bug info etc in
/var/log/messages.
After enabling KEXEC feature those info was missed.
How can we get those missed info in /var/log/messages even when
KEXEC
feature is enabled.
Any help will appreciate.
run crash on the resultant vmcore file that kexec can produce/save
and
get the
dmesg log out of it.
alternatively you can use makedumpfile with the --dump-dmesg log
Neil
Post by Dharmosoth Seetharam
Thanks and Regards
Seetharam
____________________________________________________________
Receive Notifications of Incoming Messages
Easily monitor multiple email accounts & access them with a click.
Visit http://www.inbox.com/notifier and check it out!
_______________________________________________
kexec mailing list
kexec at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/kexec
____________________________________________________________
GET FREE SMILEYS FOR YOUR IM & EMAIL - Learn more at
http://www.inbox.com/smileys
Works with AIM?, MSN? Messenger, Yahoo!? Messenger, ICQ?, Google Talk?
and most webmails
Loading...