Jump to content
ngokli

log files not rolling

Recommended Posts

I installed plastic on a new CentOS 7 server yesterday, and copied the sqlite databases and config files from an old server we are decomissioning.  Then, a few hours after starting the plasticsd service, at midnight UTC, I found these error messages in my ssh session console:

$ sudo service plasticsd start
Starting the Plastic SCM server:                                     OK
$ log4net:ERROR [RollingFileAppender] Exception while rolling file [/opt/plasticscm5/server/plastic.relevant.log.txt] -> [plastic.relevant.log.txt.20170531]
System.UnauthorizedAccessException: Access to the path is denied.
  at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00140] in <8f2c484307284b51944a1a13a14c0266>:0
  at log4net.Appender.RollingFileAppender.RollFile (System.String fromFile, System.String toFile) [0x00052] in <4bfebc64fd4743b4b376302a10c947e5>:0
log4net:ERROR [RollingFileAppender] Exception while rolling file [/opt/plasticscm5/server/plastic.debug.log.txt] -> [plastic.debug.log.txt.20170531]
System.UnauthorizedAccessException: Access to the path is denied.
  at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00140] in <8f2c484307284b51944a1a13a14c0266>:0
  at log4net.Appender.RollingFileAppender.RollFile (System.String fromFile, System.String toFile) [0x00052] in <4bfebc64fd4743b4b376302a10c947e5>:0
log4net:ERROR [RollingFileAppender] Exception while rolling file [/opt/plasticscm5/server/plastic.channelcall.log.txt] -> [plastic.channelcall.log.txt.20170531]
System.UnauthorizedAccessException: Access to the path is denied.
  at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00140] in <8f2c484307284b51944a1a13a14c0266>:0
  at log4net.Appender.RollingFileAppender.RollFile (System.String fromFile, System.String toFile) [0x00052] in <4bfebc64fd4743b4b376302a10c947e5>:0

At this point, the log files were cleared, and they continue to accumulate new messages.  There are old log files.  I couldn't find the errors above in any log file, though (even grepping all of /var/log).  I suspect they (luckily) showed up in that terminal because that's where I'd started plasticsd.

plasticd is running as the plasticscm user:

$ ps aux | grep -i 'plastic\|RSS TTY'
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     16679  0.0  0.0 111788  1476 pts/0    S    19:38   0:00 sudo -u plasticscm -b -n /opt/plasticscm5/server/plasticd --daemon
plastic+ 16681  0.0  1.1 1318244 92192 pts/0   Sl   19:38   0:01 plasticd /opt/plasticscm5/server/plasticd.exe --daemon
ngokli   16817  0.0  0.0 112652   984 pts/0    S+   20:00   0:00 grep --color=auto -i plastic\|RSS TTY

Directory permissions look okay:

$ pwd
/opt/plasticscm5/server
$ ls -ldh *.log *.log.txt . ..
drwxr-xr-x. 7 plasticscm plasticscm 8.0K Jun  1 20:03 .
drwxr-xr-x. 7 root       root         76 May 31 19:14 ..
-rw-r--r--. 1 plasticscm plasticscm    0 May 31 19:14 audit.log
-rw-r--r--. 1 plasticscm plasticscm 451K Jun  1 20:14 plastic.channelcall.log.txt
-rw-r--r--. 1 plasticscm plasticscm  16M Jun  1 20:14 plastic.debug.log.txt
-rw-r--r--. 1 plasticscm plasticscm  642 Jun  1 19:37 plastic.errors.log.txt
-rw-r--r--. 1 plasticscm plasticscm 874K Jun  1 20:14 plastic.realtime.requests.log.txt
-rw-r--r--. 1 plasticscm plasticscm 8.9M Jun  1 20:14 plastic.relevant.log.txt

SELinux is in permissive mode (so it shouldn't block anything):

$ sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

I installed plasticscm-complete-5.4.16.918-1.1, to stay as close to my old server as possible (will upgrade soon):

$ yum list installed '*plastic*'
Loaded plugins: fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirror.genesisadaptive.com
 * extras: mirror.genesisadaptive.com
 * updates: mirror.genesisadaptive.com
Installed Packages
plasticscm-certtools-mono4.noarch              3.12.1-1.1                @plasticscm-common
plasticscm-client-complete.noarch              5.4.16.918-1.1            @plasticscm-latest
plasticscm-client-core.x86_64                  5.4.16.918-2.1            @plasticscm-latest
plasticscm-client-gtk.noarch                   5.4.16.918-1.1            @plasticscm-latest
plasticscm-complete.noarch                     5.4.16.918-1.1            @plasticscm-latest
plasticscm-gnome-sharp-mono4.x86_64            2.24.0-2.1                @plasticscm-common
plasticscm-gtk-sharp-mono4.x86_64              2.12.29-2.1               @plasticscm-common
plasticscm-mono4.x86_64                        4.6.2-2.1                 @plasticscm-common
plasticscm-server-complete.noarch              5.4.16.918-1.1            @plasticscm-latest
plasticscm-server-core.noarch                  5.4.16.918-2.1            @plasticscm-latest
plasticscm-theme.noarch                        5.4.16.918-2.1            @plasticscm-latest

Sooo.... I need those log files to stick around, so where do I go from here?

Thanks!

Neal

P.S. I hit submit on this forum post, and it told me I wasn't authorized, and hitting the browser back button also showed me an unauthorized message.  Luckily, I was able to recover my message after logging in in a different window and telling the browser to resubmit the POST, but this is dangerous!

Share this post


Link to post
Share on other sites

Hi!

permissions seems to be OK.

Can you check if you can run this command? 

sudo -u plasticscm cp plastic.debug.log.txt plastic.debug.log.txt.roll1

The error is pretty clear though.

System.IO.File.Move -> Access to the path is denied

I'll try to search related issues with log4net.

Share this post


Link to post
Share on other sites

Thanks for the reply, Manu!

It seems like most of the log files started rolling properly on Saturday.  I don't think I touched anything since my original post, I've been working on other things...  I'll restart plasticsd from a terminal before midnight UTC tomorrow to see if there are any error messages (there were on Friday).

I just tried running your cp command, and of course it works now!  I don't know if it would have worked last week... :(

I'll report back if I find there are still issues.

Thanks, Manu!

Share this post


Link to post
Share on other sites

Ok! No problem.

Just let us know if you still notice issues... we can try to get connected together and check if we can see something weird.

Happy to know it's working fine :)

Share this post


Link to post
Share on other sites

Aha!  I set my ssh shell up to catch the error messages, and they showed up again:

log4net:ERROR [RollingFileAppender] Exception while rolling file [/opt/plasticscm5/server/plastic.relevant.log.txt] -> [plastic.relevant.log.txt.20170606]
System.UnauthorizedAccessException: Access to the path is denied.
  at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00140] in <8f2c484307284b51944a1a13a14c0266>:0
  at log4net.Appender.RollingFileAppender.RollFile (System.String fromFile, System.String toFile) [0x00052] in <4bfebc64fd4743b4b376302a10c947e5>:0
log4net:ERROR [RollingFileAppender] Exception while rolling file [/opt/plasticscm5/server/plastic.debug.log.txt] -> [plastic.debug.log.txt.20170606]
System.UnauthorizedAccessException: Access to the path is denied.
  at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00140] in <8f2c484307284b51944a1a13a14c0266>:0
  at log4net.Appender.RollingFileAppender.RollFile (System.String fromFile, System.String toFile) [0x00052] in <4bfebc64fd4743b4b376302a10c947e5>:0
log4net:ERROR [RollingFileAppender] Exception while rolling file [/opt/plasticscm5/server/plastic.channelcall.log.txt] -> [plastic.channelcall.log.txt.20170606]
System.UnauthorizedAccessException: Access to the path is denied.
  at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00140] in <8f2c484307284b51944a1a13a14c0266>:0
  at log4net.Appender.RollingFileAppender.RollFile (System.String fromFile, System.String toFile) [0x00052] in <4bfebc64fd4743b4b376302a10c947e5>:0
log4net:ERROR [RollingFileAppender] Exception while rolling file [/opt/plasticscm5/server/plastic.errors.log.txt] -> [plastic.errors.log.txt.20170606]
System.UnauthorizedAccessException: Access to the path is denied.
  at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00140] in <8f2c484307284b51944a1a13a14c0266>:0
  at log4net.Appender.RollingFileAppender.RollFile (System.String fromFile, System.String toFile) [0x00052] in <4bfebc64fd4743b4b376302a10c947e5>:0

It seems the log files rolled fine for three days, and then it broke again...

]$ ls -lh *log.txt*
-rw-r--r--. 1 plasticscm plasticscm 106K Jun  7 01:33 plastic.channelcall.log.txt
-rw-r--r--. 1 plasticscm plasticscm  83K Jun  3 23:59 plastic.channelcall.log.txt.20170603
-rw-r--r--. 1 plasticscm plasticscm  76K Jun  4 23:55 plastic.channelcall.log.txt.20170604
-rw-r--r--. 1 plasticscm plasticscm 162K Jun  5 23:56 plastic.channelcall.log.txt.20170605
-rw-r--r--. 1 plasticscm plasticscm 1.7M Jun  7 01:34 plastic.debug.log.txt
-rw-r--r--. 1 plasticscm plasticscm  16M Jun  3 23:59 plastic.debug.log.txt.20170603
-rw-r--r--. 1 plasticscm plasticscm  16M Jun  4 23:59 plastic.debug.log.txt.20170604
-rw-r--r--. 1 plasticscm plasticscm  16M Jun  5 23:59 plastic.debug.log.txt.20170605
-rw-r--r--. 1 plasticscm plasticscm  983 Jun  7 00:14 plastic.errors.log.txt
-rw-r--r--. 1 plasticscm plasticscm  558 Jun  3 02:36 plastic.errors.log.txt.20170603
-rw-r--r--. 1 plasticscm plasticscm  558 Jun  4 06:54 plastic.errors.log.txt.20170604
-rw-r--r--. 1 plasticscm plasticscm  558 Jun  5 11:12 plastic.errors.log.txt.20170605
-rw-r--r--. 1 plasticscm plasticscm 2.5M Jun  7 01:33 plastic.realtime.requests.log.txt
-rw-r--r--. 1 plasticscm plasticscm 694K Jun  7 01:34 plastic.relevant.log.txt
-rw-r--r--. 1 plasticscm plasticscm  11M Jun  3 23:59 plastic.relevant.log.txt.20170603
-rw-r--r--. 1 plasticscm plasticscm  11M Jun  4 23:59 plastic.relevant.log.txt.20170604
-rw-r--r--. 1 plasticscm plasticscm  11M Jun  5 23:59 plastic.relevant.log.txt.20170605

I happened to reboot the server just before this happened...  But I've stopped and started plasticsd service itself several times in the past three days.

I am able to copy a logfile as plasticscm user.

I'm wondering if it's trying to put the rolled logfile in '/' instead of '/opt/plasticscm5/server'.  '/' seems to be the working directory of the plasticsd service.  On the old server, there are three sets of plastic logfiles from seemingly-random dates in '/'.  I haven't found anything special that happened on those three dates...

I just modified my loader.log.conf to put my logfiles in '/var/log/plasticscm5' (where logs belong on linux).  Maybe specifying an absolute path will help!

 

Thanks, Manu!  I'll let you know if I am still having issues...

Neal

 

 

Share this post


Link to post
Share on other sites

By the way, you can ignore 'plastic.realtime.requests.log.txt', because it's set to roll based on size, not date.

Share this post


Link to post
Share on other sites
On 6/7/2017 at 3:44 AM, ngokli said:

I'm wondering if it's trying to put the rolled logfile in '/' instead of '/opt/plasticscm5/server'.  '/' seems to be the working directory of the plasticsd service.  On the old server, there are three sets of plastic logfiles from seemingly-random dates in '/'.  I haven't found anything special that happened on those three dates...

That's super weird! But might explain the error...

How did you manage to get the plasticd working directory?

I'm wondering if log4net is having a log for themselves so we can check the rolling destination path, I'll search for it.

It's strange it's working something and sometimes not.

Thank you for your time debugging this! To be honest with you this is the first time I face this behavior.

Share this post


Link to post
Share on other sites

Hi, did you guys figure this one out?

I'm seeing the same issue now. "sudo -u plasticscm" has no problems with permissions but I get these:

 log4net:ERROR [RollingFileAppender] Exception while rolling file [/opt/plasticscm5/server/plastic.debug.log.txt] -> [plastic.debug.log.txt.20171107]
System.UnauthorizedAccessException: Access to the path is denied.
  at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00140] in <8f2c484307284b51944a1a13a14c0266>:0
  at log4net.Appender.RollingFileAppender.RollFile (System.String fromFile, System.String toFile) [0x00052] in <4bfebc64fd4743b4b376302a10c947e5>:0

This definitely started to happen after a restart (restarted both service and then the whole server machine though).

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×